Patience Is A Virtue: Wait Scripts And Visual Web Ripper

By Author: Tracy Morgan
Total Articles: 20
Comment this article

A problem faced by web data extractor software is similar to one faced by automatic phone dialers.

When automatic phone dialers first became available, their usefulness was limited due to the delay in receiving a dial tone when the phone line was initially engaged. By the time the line was ready, the dialer had already entered a few digits, causing it to consistently fail. A similar premature extraction can occur when scraping data from web sites.
To overcome this, a Wait Script is necessary when an AJAX Callback is unable to successfully retrieve data due to the webmaster's use of a transitional word or phrase (typically a blinking “Loading...”) to tell the user the content is rendering. Visual Web Ripper confuses this for the actual data and extracts that unwanted text.

Remember good things come to those who wait. Follow these steps to have Visual Web Ripper extract the data you want.

Note: A working knowledge of OOP, especially in C# and VB.NET is EXTREMELY helpful.
To begin your Wait Script, navigate to your Options menu, click Action, click AJAX or javascript and click Script Wait Condition.

A ...
... window pops up, allowing you to manipulate premade code (either in C# or VB.NET). If the code is inaccessible, check Enable wait script.

Your code should look like this at the start:

1. public static bool IsAjaxCallCompleted(WrWaitScriptArguments args)   
2. {
3. try
4. {
5. //Placeyour condition code here.
6. //This example waits until the old content has been replaced with new content,
7. //but waits a maximum of 3seconds.

8. if (args.SecondsWaited>3)
9. return true ;
10.
11. if (!args.OldContent.Equals(args.NewContent))
12. return true ;

13. return false ;
14. }
15. catch (Exceptionexp)
16. {
17. args.WriteDebug(exp.Message);
18. return true ;
19. }
20. }
Be aware of some requirements with this solution:
All Wait Scripts must have one method (IsAjaxCallCompleted)
That method must have that exact name, so only change within the body.
The method must return true when the AJAX call has been completed and false otherwise.

The “loading” example would look like this (within the body):
1. tr
2. {
3. if (args.SecondsWaited>3)
4. return true ;
5.
6. if (!args.OldContent.Equals(args.NewContent)&&!args.NewContent.Equals( "Loading..." ))
7. return true ;
8.
9. return false ;
10. }
11. catch (Exceptionexp)
12. {
13. args.WriteDebug(exp.Message);
14. return true ;
15. }
16. }
17. }

Cascading Drop-Down Menus
A cascading drop-down menu is another hurdle that requires special handling when using Visual Web Ripper. Utilize this code to manipulate cascading drop-down menus{
1. if(args.SecondsWaited>10)
2. returntrue;
3.
4. if(!args.OldContent.Equals(args.NewContent) && !args.NewText.StartsWith("["))   
5. return true;
6.
7. returnfalse;
8. }
9.         catch (Exception exp)   
10.         {   
11.             args.WriteDebug(exp.Message);   
12.             return true;   
13.         }   
14.     }   
15. }  
Notes: A FormField element will be needed for every drop-down menu. VWR will wait for every AJAX element to change.

As web sites include an increased set of functionality to enhance user experience, this same functionality requires more powerful web data extraction software. Visual Web Ripper capably delivers in cases like this.

Author:
For more information about Best Web Scraping Software Please visit www.visualwebripper.comA problem faced by web data extractor software is similar to one faced by automatic phone dialers.

When automatic phone dialers first became available, their usefulness was limited due to the delay in receiving a dial tone when the phone line was initially engaged. By the time the line was ready, the dialer had already entered a few digits, causing it to consistently fail. A similar premature extraction can occur when scraping data from web sites.
To overcome this, a Wait Script is necessary when an AJAX Callback is unable to successfully retrieve data due to the webmaster's use of a transitional word or phrase (typically a blinking “Loading...”) to tell the user the content is rendering. Visual Web Ripper confuses this for the actual data and extracts that unwanted text.

Remember good things come to those who wait. Follow these steps to have Visual Web Ripper extract the data you want.

Note: A working knowledge of OOP, especially in C# and VB.NET is EXTREMELY helpful.
To begin your Wait Script, navigate to your Options menu, click Action, click AJAX or javascript and click Script Wait Condition.

A window pops up, allowing you to manipulate premade code (either in C# or VB.NET). If the code is inaccessible, check Enable wait script.

Your code should look like this at the start:

1. public static bool IsAjaxCallCompleted(WrWaitScriptArguments args)   
2. {
3. try
4. {
5. //Placeyour condition code here.
6. //This example waits until the old content has been replaced with new content,
7. //but waits a maximum of 3seconds.

8. if (args.SecondsWaited>3)
9. return true ;
10.
11. if (!args.OldContent.Equals(args.NewContent))
12. return true ;

13. return false ;
14. }
15. catch (Exceptionexp)
16. {
17. args.WriteDebug(exp.Message);
18. return true ;
19. }
20. }
Be aware of some requirements with this solution:
All Wait Scripts must have one method (IsAjaxCallCompleted)
That method must have that exact name, so only change within the body.
The method must return true when the AJAX call has been completed and false otherwise.

The “loading” example would look like this (within the body):
1. tr
2. {
3. if (args.SecondsWaited>3)
4. return true ;
5.
6. if (!args.OldContent.Equals(args.NewContent)&&!args.NewContent.Equals( "Loading..." ))
7. return true ;
8.
9. return false ;
10. }
11. catch (Exceptionexp)
12. {
13. args.WriteDebug(exp.Message);
14. return true ;
15. }
16. }
17. }

Cascading Drop-Down Menus
A cascading drop-down menu is another hurdle that requires special handling when using Visual Web Ripper. Utilize this code to manipulate cascading drop-down menus{
1. if(args.SecondsWaited>10)
2. returntrue;
3.
4. if(!args.OldContent.Equals(args.NewContent) && !args.NewText.StartsWith("["))   
5. return true;
6.
7. returnfalse;
8. }
9.         catch (Exception exp)   
10.         {   
11.             args.WriteDebug(exp.Message);   
12.             return true;   
13.         }   
14.     }   
15. }  
Notes: A FormField element will be needed for every drop-down menu. VWR will wait for every AJAX element to change.

As web sites include an increased set of functionality to enhance user experience, this same functionality requires more powerful web data extraction software. Visual Web Ripper capably delivers in cases like this.

Author:
For more information about Best Web Scraping Software Please visit www.visualwebripper.com