Jump to content

Platinum level Jagex recognized fansite

Toggle shoutbox ColorWars ShoutBox

2003 Veteran : (Jul 31,2014 12:18) Grats :) Keep it up twiz!
Zyneste : (Jul 31,2014 4:17) Congrats, keep it up then! ;)
twiztedj : (Jul 31,2014 4:07) Just now signed up but have been using the quest guides. Just hit 43 prayer and can us full rune now!
Fading Rain : (Jul 30,2014 23:17) arma isnt that bad in both legacy or eoc... thought it was going to be much worse XD
Fading Rain : (Jul 30,2014 23:05) always worth a shot, and just got stream back up XD
2003 Veteran : (Jul 30,2014 22:50) Seriously tho, if I made an ironman account, blogged and made videos, would anyone watch it? Lol
Fading Rain : (Jul 30,2014 22:42) If people are bored, going to solo arma and stream to twitch. twitch.tv/fading_rain
McSwindler : (Jul 30,2014 20:21) Just finished ME1 on OS, looked at the guide for ME2 and began crying in feeble position.
2003 Veteran : (Jul 30,2014 20:03) Smarty pants aren't we Gator? I like you. Lol :)
DixR4Chixx : (Jul 30,2014 19:47) Ian W I'll do it with you later?
DixR4Chixx : (Jul 30,2014 19:47) Is anyone free to help me with The Shield of Arrav quest later?
The Gatorboy : (Jul 30,2014 19:34) Tony Stark would, he's an Iron Man superfan
2003 Veteran : (Jul 30,2014 19:31) If I was to make an oldschool Ironman account and start a blog on here, who'd follow it? :P
The Gatorboy : (Jul 30,2014 19:30) BABY COME BACK! YOU CAN BLAME IT ALLLL ONNNN SLIVEERRRRRRR
The Gatorboy : (Jul 30,2014 19:24) noooo don't. please come back online, you are never online anymore :(
Shaleclaw : (Jul 30,2014 19:22) I am the Great Shaleclaw Houdini! I can make myself disappear!
The Gatorboy : (Jul 30,2014 19:16) SHALECLAW!!!! I HAVENT SEEN YOU IN SO LONG!!!
Shaleclaw : (Jul 30,2014 19:09) Dubstep in the Shoutbox. Just when you think you've seen it all... :P
The Gatorboy : (Jul 30,2014 18:51) why wake up to BEEP BEEP BEEP BEEP BEEP and MEOOOOOOOOOW MEOOOOOOOOOOOOOW when you can wake up to FIRE UP YOUR LOUD ANOTHER ROUND OF SHOTS...........TURN DOWN FOR WHAT!!!!! WUB WUB WUB WUB WUB WUB WUB WUB WUB....and MEOOOOOOOOOOOOW MEOOOOOOOOOOOOOOOOOW
The Gatorboy : (Jul 30,2014 18:49) like first

Grand Exchange Crawler (.NET Source)


This topic has been archived. This means that you cannot reply to this topic.
4 replies to this topic

#1 Guest_Pinch d_*

Guest_Pinch d_*
  • Guests
Click to view battle stats

Posted 30 September 2012 - 04:12 AM

I've started a project to create a Grand Exchange Analysis program, whereby you could manage your stock (logs/ores/rares/etc) and the financial gains (or loss) day by day, for any number of items and quantities. While the Grand Exchange itself provides all of the information, it doesn't allow you to make any calls without calculating the profit margin yourself.

Projected Features:
Given an input of the item you currently have, and quanitity, upon startup the program will grab the latest GE prices, and calculate your profit from time of purchase (if known, and within 180days)
Calculate graph curve, and attempt rough prediction of item value fluctuation
Update grand exchange list (remotely hosted itemdb, or locally built, which at this moment in time is data usage heavy)
Slide away interface
Item search combining wikia listing & current GE Price data in result
Open source, implement via webserver or as a local program


At this stage, the grand exchange item crawling feature is 100% working (needs revising, but scans reasonably well), and you'll find that at the end of the thread!

For some of you out there who may enjoy programming, the projects current code would be great for indexing the GE, as apparently the item numbers change from time to time, and being able to reliably check a range of indexes for change could be very useful.

To compile yourself:
-Listview Control (3 columns, itemindex itemname itemurl)
- 2 x TextBox (1 & 2) - textbox1 is starting index, textbox2 is final index
Reference to mshtml.dll (7.5)


Imports System
Imports System.IO
Imports System.Net
Imports System.Text
Imports mshtml

Public Class itemdbcrawler


    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        Dim itemindex As Integer = CInt(TextBox2.Text)
        Dim itemcount As Integer = CInt(TextBox1.Text)
        Dim baseurl As String = "http://services.runescape.com/m=itemdb_rs/viewitem.ws?obj="
        Dim countedurl As String

        While itemcount < itemindex
            countedurl = baseurl & itemcount.ToString
            Dim request As WebRequest = _
              WebRequest.Create(countedurl)
            request.Credentials = CredentialCache.DefaultCredentials
            Dim response As WebResponse = request.GetResponse()
            Console.WriteLine(CType(response, HttpWebResponse).StatusDescription)
            Dim dataStream As Stream = response.GetResponseStream()
            Dim reader As New StreamReader(dataStream)
            Dim responseFromServer As String = reader.ReadToEnd()
            Dim doc As IHTMLDocument2 = New HTMLDocumentClass()
            doc.write(New Object() {responseFromServer})
            doc.close()
            reader.Close()
            response.Close()
            Dim ContentStr As String = metacapture(doc)
            If Not ContentStr.Contains("role-playing") Then
                Dim fields() As String
                fields = Split(ContentStr, ",")
                Dim Index As String = itemcount.ToString
                Dim Name As String = fields(1)
                Dim Url As String = countedurl.ToString
                ListViewAddItem(Index, Name, Url)
            End If
            itemcount += 1
        End While
        MessageBox.Show("Crawl Complete")
    End Sub

    Private Function metacapture(ByVal htmldoc As IHTMLDocument) As String

        If (htmldoc IsNot Nothing) Then
            Dim Elems As IHTMLElementCollection
            Dim Errors As IHTMLElementCollection
            Dim WebOC As IHTMLDocument = htmldoc
            Errors = WebOC.getElementsByTagName("div")
            For Each elem As IHTMLElement In Errors
                Dim NameStr As String = elem.getAttribute("id")
                If ((NameStr IsNot Nothing) And (NameStr.Length <> 0)) Then
                    If NameStr.ToLower().Equals("errorcontent") Then
                        Return "0"
                    Else
                        Elems = WebOC.getElementsByTagName("meta")
                        For Each elem2 As IHTMLElement In Elems
                            Dim NameStr2 As String = elem2.getAttribute("name")
                            If ((NameStr2 IsNot Nothing) And (NameStr2.Length <> 0)) Then
                                If NameStr2.ToLower().Equals("keywords") Then
                                    Dim ContentStr As String = elem2.getAttribute("content")
                                    Dim returnstring As String
                                    Dim fields() As String
                                    fields = Split(ContentStr, ",")
                                    returnstring = (WebOC.url.ToString() & "," & fields(3))
                                    Return returnstring
                                End If
                            End If
                        Next
                    End If
                End If
            Next
        End If
    End Function

    Private Delegate Sub ListViewAddItem_delegate(ByVal i_index As String, ByVal i_name As String, ByVal i_url As String)

    Private Sub ListViewAddItem(ByVal i_index As String, ByVal i_name As String, ByVal i_url As String)
        Dim myArgs(2) As Object
        myArgs(0) = i_index
        myArgs(1) = i_name
        myArgs(2) = i_url
        If Me.ListView1.InvokeRequired Then
            Dim d As New ListViewAddItem_delegate(AddressOf ListViewAddItem)
            Me.ListView1.BeginInvoke(d, myArgs)
        Else
            Dim lvi As New ListViewItem(i_index)
            lvi.SubItems.Add(i_name)
            lvi.SubItems.Add(i_url)
            ListView1.Items.AddRange(New ListViewItem() {lvi})
            ListView1.Update()
        End If
    End Sub
End Class


#2 Claudia

Claudia

    ♥

  • Established Members
  • PipPipPipPipPip
  • 556 posts
Click to view battle stats

User's Awards

 Â 

Posted 30 September 2012 - 06:56 AM

That sounds pretty neat-o. A little too smart for me, but neat-o nonetheless.

Best of luck with this awesome project.

Posted Image


#3 Symbolic

Symbolic

    Influential Iron Dragon

  • Advanced Members
  • 561 posts
Click to view battle stats

User's Awards

 Â   Â 

Posted 30 September 2012 - 08:53 AM

Looks pretty cool. Good luck with your project and keep us updated!
Posted Image

#4 Guest_Pinch d_*

Guest_Pinch d_*
  • Guests
Click to view battle stats

Posted 30 September 2012 - 09:05 PM

Moving along, across the next week, ill be building the database using the above code, with slight modifications to the 2nd node 'for each', to collect description, Item Icon URL (saving upwards of 50k icons would be killer, better to load OTF from a personal view) & price. Im contemplating making the initial database dump (raw container) a tab delimited txt file, then use my local SQL server to build a proper database.

The next step after that, is to rework the existing code into a deployable database creation tool, and depending on how complex the database needs to be, attempt to add support for various other database formats.

Heres how the GEC is looking so far!

Attached File  currentprog.jpg   184.24KB   19 downloads

#5 Guest_Pinch d_*

Guest_Pinch d_*
  • Guests
Click to view battle stats

Posted 03 October 2012 - 05:36 PM

so far 00000-10000 have been indexed, and progress is going smooth.

Currently working on a Farming buddy, an event driven alarm, press a hotkey of your choice, a dialog box appears, giving you a selection of farming locations & plots, and what youre planting, and a selection of per-stage reminders or expected fully grown time. Because honestly, micromanaging farming patches can be a pain, now it'll be a whole lot easier (:

if you have any suggestions to what you'd find handy helpful, im more than happy to work a solution, and provide code and in the next 2 days a code project link to compiled .exe's (.net 3 or above required)