[TBA]: Team info update?

[smcmahon]

Greetings TBA. I'm with Team 2656, entering our second season of FRC. I was just looking at our page on the TBA website (which is awesome BTW) and a thought occurred to me... We've updated our team name/nick name on TIMS and added URL's there for our website and blog. I'd like to see this information mirrored on TBA, but I wasn't sure how/if I could update that. I assume that you all have something that syncs data with FIRST/TIMS? If so, then I guess we'll be all good after the next update and I won't worry about it any longer. If not, let me know and I'll be happy to submit further information for Team 2656. Thanks. I really like what ya'll and doing at TBA and we've got a link to our page there on the main page of our website. Please, keep up the great work!

[Greg Marra]

I just went to run an update of the team data, and it seems that FIRST is doing something to block scripts from downloading team data. I am not sure what is going on, but (https://my.usfirst.org/myarea/index.lasso?page=teaminfo&event_type=FRC&team=1388) gives me different source code depending on if I open it in Chrome or wget it, and I can't open the URL at all from my PHP scripts that worked last spring.

I will investigate this further, but we will do a complete team data update before ship date.

Thanks!

[EHaskins]

I just went to run an update of the team data, and it seems that FIRST is doing something to block scripts from downloading team data. I am not sure what is going on, but (https://my.usfirst.org/myarea/index.lasso?page=teaminfo&event_type=FRC&team=1388) gives me different source code depending on if I open it in Chrome or wget it, and I can't open the URL at all from my PHP scripts that worked last spring.

I will investigate this further, but we will do a complete team data update before ship date.

Thanks!

I'm curious if they were having an issue with abuse.

[RyanCahoon]

Sounds like user agent sniffing to me. Could be hacked by setting the user agent (i.e. in PHP as described here (http://www.seopher.com/articles/how_to_change_your_php_user_agent_to_avoid_being_b locked_when_using_curl)) in whatever scripts are used, but it seems to me that a site as prominent in the FIRST community as TBA should be able to work out something a little more "over the table" with FIRST.

--Ryan

[Matt Keller]

The issue is that they switched TIMS data over to a secure site, and PHP lacked any SSL support until 4.3.0, and ever since it has been rather flaky. As an alternative without fooling with any libraries, you may be able to use fsockopen() to create a SSL or TLS socket. More than likely your host has heavy restrictions on socket-related functions, especially if you are on a shared hosting plan.

When they made the switch last season, I immediately updated my scripts to utilize the cURL library, which works beautifully in this situation. Many hosts now support it and I definitely recommend going this route.

Here would be a simple cURL replacement for file_get_contents():

<?php
$ch = curl_init();
$timeout = 5; // set to zero for no timeout
curl_setopt ($ch, CURLOPT_URL, 'http://example.com');
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$file_contents = curl_exec($ch);
curl_close($ch);

// display file
echo $file_contents;
?>



and a replacement for file():

<?php
$ch = curl_init();
$timeout = 5; // set to zero for no timeout
curl_setopt ($ch, CURLOPT_URL, 'http://example.com');
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$file_contents = curl_exec($ch);
curl_close($ch);
$lines = array();
$lines = explode("\n", $file_contents);

// display file line by line
foreach($lines as $line_num => $line) {
echo "Line # {$line_num} : ".htmlspecialchars($line)."<br />\n";
}
?>


If you are unable to use these methods and can not get anything to work, I am more than willing to help out.

Hope this helps!

[jtdowney]

I'm curious if they were having an issue with abuse.

I think a more likely explanation is they are trying to optimize the page for the user agent it detects. Their site has to handle a large amount of traffic and intentionally blocking screen scraping seems like a wasted effort to me.

[Greg Marra]

I think a more likely explanation is they are trying to optimize the page for the user agent it detects. Their site has to handle a large amount of traffic and intentionally blocking screen scraping seems like a wasted effort to me.

I don't think they do any page optimization based on client. FIRST's TIMS system is pretty old and hasn't seen any substantial updates in the last few years. What would be the best is if they implemented an actual API that provided structured XML so that teams could more easily write applications that utilized FIRST's data.

Ultimately, I feel that FIRST underutilizes their website, and could do a lot to promote and help the community. Potential sponsors and people looking for "official" info need a very different experience than students and mentors currently on teams, and I don't think FIRST does a very good job catering to the latter group right now.

[jtdowney]

I don't think they do any page optimization based on client. FIRST's TIMS system is pretty old and hasn't seen any substantial updates in the last few years. What would be the best is if they implemented an actual API that provided structured XML so that teams could more easily write applications that utilized FIRST's data.

Ultimately, I feel that FIRST underutilizes their website, and could do a lot to promote and help the community. Potential sponsors and people looking for "official" info need a very different experience than students and mentors currently on teams, and I don't think FIRST does a very good job catering to the latter group right now.

If FIRST had an API to retrieve public information from TIMS it would probably make many lives easier. However from my understanding they outsource almost all of that stuff to a company, 4x media or something similar.

[Chris Marra]

If FIRST had an API to retrieve public information from TIMS it would probably make many lives easier. However from my understanding they outsource almost all of that stuff to a company, 4x media or something similar.
4fx, IIRC, is the company that has been responsible for helping with competition control systems in the past few years since they stopped contracting Hatch Technologies. They wrote the match pairing algorithms and the software used at the regional to handle rankings and brackets, which exports the data to the FIRST website. This might export awards to TIMS, but TIMS is an older system than 4fx, and possibly than Hatch.

[Nibbles]

I have no problems with PHP 5.2.6 or wget 1.11.4 irrespective of user agent (the user_agent setting and -U argument respectively). (I forgot how much whitespace they use though.) Try again?