Programming language for basic data gathering from websites

Chomeur

I'd like to be able to write a programme that checks simple things on certain websites. I have Microsoft Visual Studio and can write Visual Basic and I can use a WebBrowser control in that to access a web page and examine its Document properties. But I'm not sure that this is the right way to go about it at all. After all web pages are not written in Visual Basic.

Is there a different language I should download and learn?

esuhl

There's a book called Webbots, Spiders and Screen Scrapers that explains how to do the kind of thing you're looking for using PHP and CURL.

nzseries1

In saying that, if it's only for a hobby, you should be fine with what you've described. I've done exactly that using Microsoft Visual Studio and C# and a WebBrowser.

Turning off automatic loading of images in Internet Explorer's settings can be a good way to prevent downloading of unnecessary stuff.

ehlo

From the thread title alone I would have seconded esuhl's post about cURL and php. However there isn't really a best programming language for this and it makes sense to simply continue with the language you already have an experience in .

I have very little experience in visual basic however using a webbrowser control may not be the best thing to use (depending on what exactly you want to do).

If you are getting a page and then just displaying a section of it then a webbrowser control makes sense. However if you are manipulating the data and just need the text and not the styling or images then the webbrowser control will also fetch these creating a lot of overhead.

I imagine there are a lot of http / HTML libraries for visual basic that will probably make this easier for you. You're probably get much better responses asking on a visual basic / Microsoft programming forum.
Another thing, have you checked to see if the site has RSS/XML/some development API that makes it easier to get the data then playing with the html viewable output?

Again, as you haven't gone into specifics it could be there is already a program that will do what you want or perhaps a firefox addon.

Chomeur

This is only really for occasional things that I like to check online and which I'd like to automate. Nothing professional.

I put a WebBrowser control in a form just to be able to get a reference to the page. No doubt there's a way to get a reference to it without the webbrowser. But anyway the hierarchy of properties seemed a nightmare to navigate and I was unable to find what I was looking for. I'll try asking on a VB forum.

RobTang

The webbrowser control just allows you to have an instance of IE running under a .net forms project so you can have html content, it doesn't do what your really looking for.

http://msdn.microsoft.com/en-us/library/system.net.webclient.aspx

WebClient is probably what your looking for, you still need to parse the returned data yourself however, or get another lib to do it for you.

There are plenty of existing scraping libs in various languages however which would be a better bet.