I always wondered how people use different online platforms. I wanted to follow the prominent figures I found on Twitter and LinkedIn. But it’s a hassle finding people on the internet.
I used to Google their names to see where else they were active. This approach works but is still manual. It’ll be a nightmare if I want to automate any part of this.
Say you want to build a database of professionals in your domain. I wouldn’t google them individually.
There are paid services to do the job. But I wouldn’t use them either. Why would I pay for a third-party service while this little Python does it for free?
The tool’s Sherlock.
Sherlock is a Python project. It helps us look up various social media platforms for profiles. You could look up people by their usernames on LinkedIn, Twitter, Instagram, Medium, etc.
Getting ready to gather social media profiles.
Before using the tool, we must install it. You can use the following script to install it in a virtual environment. Technically we’re not installing Sherlock. We clone Sherlock’s repository and only install its dependencies.
Note you could also install it using Poetry to avoid the pitfalls of using virtual environments directly.
# Clone the Sherlock GitHub repository $ git clone https://github.com/sherlock-project/sherlock.git # Change working directory $ cd sherlock # Create and activate a virtualenv. $ virtualenv env --python=3.9 $ source env/bin/activate # $ source env/Scripts/activate # on Windows # Install dependencies $ pip install -r requirements.txt
Finding people’s social media presence across the internet.
I often want to follow professionals in my domain on many platforms. But I didn’t have an easy way to find what other platforms they were on and what would be their profile link.
After installing Sherlock, we can quickly get all this information in just a single command.
The following command will show all my social media presence.
python sherlock thuwarakesh
As you can see, within a minute, it pulls my various social media profile links.
You could also lookup up many usernames at once. Just pass usernames separated by a space.
python sherlock thuwarakesh thuwalive
Avoid banning when scraping the internet.
Under the hood, Sherlock scrapes the internet. You should be aware of all the governing laws in your region before you use this tool to get social media links.
Some people may prefer to stay anonymous when they do automated stuff on the internet. If so, you can configure Sherlock to use tor.
Tor is an open-source network for anonymous browsing. You can download and install it from Tor’s official website.
Ubuntu users can install Tor with the following command.
sudo apt install tor
Once you have tor installed, you can pass in the -t flag to the Sherlock command. Now you’re finding social media profiles while staying anonymous.
python sherlock thuwarakesh -t
Another technique to avoid banning is to use a shorter timeout. You can specify a custom timeout with the -timeout argument in seconds.
python sherlock thuwarakesh --timeout 3
Lastly, a popular approach to use when scraping is a proxy. You can use a proxy to hide your identity on the internet. If you have a proxy, you can configure Sherlock to use the proxy as follows:
python sherlock thuwarakesh --proxy <Your proxy URL>
If you don’t have a proxy I highly recommend you use one. Check out NordVPN (sponsored) because it has been trusted by many well-known brands.