City Ceases Darknet Machine Learning Project

In October 2016, the Boston Police Department filed a request for access to software and technology that raised a minor difference in the community. Technically, the Boston Police Department filed a Request for Proposals. As suggested by the name, the Department requested proposals from outside security organizations and individuals In an email, Police Chief William Evans explained the RFP as one that would “aid the Department in more effectively analyzing publicly available information found on the Internet.”

Chief Evans continued with the explanation—the Department needed resources to obtain, store, and handle internet data. The data in question not only consisted of what one might expect from a police department; the Requests for Proposals laid out a specific skillset That reached beyond the inner layers of the internet. Participants needed skills in various sectors, starting with social networks, especially Twitter, and ending with darknet forums and marketplaces.

In literal terms, the request for proposals aimed to “acquire technology and services that supported the identification, collection, synthesis, analysis, and investigation of threat information present within real-time open source and social media platforms,” the advertisement summarized. Evans further clarified that the goal included the ability to use a stream of data to enhance situational awareness.

Respondents needed to meet a certain criteria of skills, as mentioned before. The darknet skills, though seemed most challenging. For instance, the “data collection” section listed 38 separate “solutions” such as the following:

Solution shall be able to query content from the “surface of the web” (e.g. Internet content indexed by most popular search engines),”deep web” (e, g. Internet content that is not indexed by popular search engines, some of which may be found in closed forums), and “dark web” (e.g. Internet content that is not indexed and is found within encrypted networks, such as TOR, requiring special web browsers and security protocols).

And still in the collection section:

Solution shall provide the ability to apply non-detectable web crawlers to query for content; Solution shall be able to access dark websites securely using TOR while hiding the user’s IP address and identifiable information using methods such as automated proxy detention and scoring, anonymizers, and virtual identities.

In section eight on page 34, the Department went into even greater detail. Again, regarding the collection of data, the application required the applicants to go into great detail regarding techniques and procedures. To win the award or contract, one must:

1. Explain the method your proposed solution will query data on the darknet.

2. Explain your proposed solution method of adopting machine learning.

3. How will your solution assist users of data matching XYZ parameters?

4. Explain the method the solution will save select darknet data.

Those examples stood out but hardly made a dent in the list of documentations, techniques, and procedures. However, companies began applying with qualifying credentials. What once was just a ludicrous myth to some creepy closer to a terrifying readily.

The Department heard no response from their superiors until 2017. Superintendent Paul Fitzgerald recommended that Chief William Evans table the project for the time being. The idea was good, but many of the candidates were overqualified for the position. Behind that, he said that he saw no need for an invasion of privacy at that scale. However, he hopes to see a similar project in the future—with citizen’s privacy in mind.