Web Scraping


So I work at basically a marketing company with a front of being a news company. For work lately I have been making a few web scrapers. These scrapers go onto our competitions sites looks at all of the users and copies their contact info. We then use those as leeds and sell our services to them at a lower price than the other guys. Now I do not feel good about this, for the fact that its all online marketing and I don't like online marketers, I also don't like to take things from other people. So I am in kind of a weird place with it. In the end of the day I get payed for it, so I do my job.


So I have some code that I put up on github that uses curl to go and do all of this stuff. What you do is go onto a site and look at all of the elements on the page, and come up with what info you want to copy, take that info and populate the file that I have made and your done. Just use PHP on your terminal and run that shit! Here is a video

The end

So now you can go and programmatically in a way steal data, now this is not really stealing because it is public info and you should be okay to copy thing. It's when you make it copy thousands a records an hour that it might be called stealing. I am not in law so I do not know. But its a kind of shitty person thing to do, but I guess people still do it. 


Popular posts from this blog

OpenSCAD Tips and Tricks

Rounded corners in openScad

m31 finger magnet rejection