I need some help with my web crawler exercise.I should write a crawler for saving the first pages of some websites and all of their content in a MySql database. I am using xampp MySql database. Here is my code:
<?php
$main_url="webpage";
$str = file_get_contents($main_url);
// Gets Webpage Title
if(strlen($str)>0)
{
$str = trim(preg_replace('/\s+/', ' ', $str)); // supports line breaks inside <title>
preg_match("/\<title\>(.*)\<\/title\>/i",$str,$title); // ignore case
$title=$title[1];
}
// Gets Webpage Description
$b =$main_url;
@$url = parse_url( $b );
@$tags = get_meta_tags($url['scheme'].'://'.$url['host'] );
$description=$tags['description'];
// Gets Webpage Internal Links
$doc = new DOMDocument;
@$doc->loadHTML($str);
$items = $doc->getElementsByTagName('a');
foreach($items as $value)
{
$attrs = $value->attributes;
$sec_url[]=$attrs->getNamedItem('href')->nodeValue;
}
$all_links=implode(",",$sec_url);
//Gets Webpage images
require_once('C:\xampp\htdocs\simple_html_dom.php');
require_once('C:\xampp\htdocs\url_to_absolute.php');
$url='webpage'
$html=file_get_html('webpage');
foreach($html->find('img') as $element) {
echo url_to_absolute($url, $element->src), "\n";
}
$images=
// Store Data In Database
$host="localhost";
$username="root";
$password="";
$databasename="db";
$connect=mysql_connect($host,$username,$password);
$db=mysql_select_db($databasename);
mysql_query("insert into webpage_details values('$main_url','$title','$description','$all_links','$images')");
?>
I should save everything from a home page, i have a problem saving the images. Any ideas? Are there any other items I need to save?
Thanks!