.. _changelog:

=========

Changelog

=========

1.3.5 (**2012-XX-XX**)

----------------------

:status: in-progress

:branch: beta

news

++++

- use ext_json for json module

- unified annotation view with file source view

- notification improvements, better inbox + css

- #419 don't strip passwords for login forms, make rhodecode 

  more compatible with LDAP servers

- Added HTTP_X_FORWARDED_FOR as another method of extracting 

  IP for pull/push logs. - moved all to base controller  

- #415: Adding comment to changeset causes reload. 

  Comments are now added via ajax and doesn't reload the page

- #374 LDAP config is discarded when LDAP can't be activated

fixes

+++++

- fixed dev-version marker for stable when served from source codes

- fixed missing permission checks on show forks page

- #418 cast to unicode fixes in notification objects

- #426 fixed mention extracting regex

1.3.4 (**2012-03-28**)

----------------------

news

++++

- Whoosh logging is now controlled by the .ini files logging setup

- added clone-url into edit form on /settings page

- added help text into repo add/edit forms

- created rcextensions module with additional mappings (ref #322) and

  post push/pull/create repo hooks callbacks

- implemented #377 Users view for his own permissions on account page

- #399 added inheritance of permissions for users group on repos groups

- #401 repository group is automatically pre-selected when adding repos 

  inside a repository group

- added alternative HTTP 403 response when client failed to authenticate. Helps 

  solving issues with Mercurial and LDAP

- #402 removed group prefix from repository name when listing repositories 

  inside a group

- added gravatars into permission view and permissions autocomplete

- #347 when running multiple RhodeCode instances, properly invalidates cache 

  for all registered servers

fixes

+++++

- fixed #390 cache invalidation problems on repos inside group

- fixed #385 clone by ID url was loosing proxy prefix in URL

- fixed some unicode problems with waitress

- fixed issue with escaping < and > in changeset commits

- fixed error occurring during recursive group creation in API 

  create_repo function

- fixed #393 py2.5 fixes for routes url generator

- fixed #397 Private repository groups shows up before login

- fixed #396 fixed problems with revoking users in nested groups

- fixed mysql unicode issues + specified InnoDB as default engine with 

  utf8 charset

- #406 trim long branch/tag names in changelog to not break UI

1.3.3 (**2012-03-02**)

----------------------

news

++++

fixes

+++++

- fixed some python2.5 compatibility issues 

- fixed issues with removed repos was accidentally added as groups, after

  full rescan of paths

- fixes #376 Cannot edit user (using container auth)

- fixes #378 Invalid image urls on changeset screen with proxy-prefix 

  configuration

- fixed initial sorting of repos inside repo group

- fixes issue when user tried to resubmit same permission into user/user_groups

- bumped beaker version that fixes #375 leap error bug

- fixed raw_changeset for git. It was generated with hg patch headers

- fixed vcs issue with last_changeset for filenodes

- fixed missing commit after hook delete

- fixed #372 issues with git operation detection that caused a security issue 

  for git repos

1.3.2 (**2012-02-28**)

----------------------

news

++++

fixes

+++++

- fixed git protocol issues with repos-groups

- fixed git remote repos validator that prevented from cloning remote git repos

- fixes #370 ending slashes fixes for repo and groups

- fixes #368 improved git-protocol detection to handle other clients

- fixes #366 When Setting Repository Group To Blank Repo Group Wont Be 

  Moved To Root

- fixes #371 fixed issues with beaker/sqlalchemy and non-ascii cache keys 

- fixed #373 missing cascade drop on user_group_to_perm table

1.3.1 (**2012-02-27**)

----------------------

news

++++

fixes

+++++

- redirection loop occurs when remember-me wasn't checked during login

- fixes issues with git blob history generation 

# -*- coding: utf-8 -*-

"""

    rhodecode.controllers.admin.settings

    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    settings controller for rhodecode admin

    :created_on: Jul 14, 2010

    :author: marcink

    :copyright: (C) 2010-2012 Marcin Kuzminski <marcin@python-works.com>

    :license: GPLv3, see COPYING for more details.

"""

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import logging

import traceback

import formencode

import pkg_resources

import platform

from sqlalchemy import func

from formencode import htmlfill

from pylons import request, session, tmpl_context as c, url, config

from pylons.controllers.util import abort, redirect

from pylons.i18n.translation import _

from rhodecode.lib import helpers as h

from rhodecode.lib.auth import LoginRequired, HasPermissionAllDecorator, \

    HasPermissionAnyDecorator, NotAnonymous

from rhodecode.lib.base import BaseController, render

from rhodecode.lib.celerylib import tasks, run_task

from rhodecode.lib.utils import repo2db_mapper, invalidate_cache, \

    set_rhodecode_config, repo_name_slug

from rhodecode.model.db import RhodeCodeUi, Repository, RepoGroup, \

    RhodeCodeSetting

from rhodecode.model.forms import UserForm, ApplicationSettingsForm, \

    ApplicationUiSettingsForm

from rhodecode.model.scm import ScmModel

from rhodecode.model.user import UserModel

from rhodecode.model.db import User

from rhodecode.model.notification import EmailNotificationModel

from rhodecode.model.meta import Session

log = logging.getLogger(__name__)

class SettingsController(BaseController):

    """REST Controller styled on the Atom Publishing Protocol"""

    # To properly map this controller, ensure your config/routing.py

    # file has a resource setup:

    #     map.resource('setting', 'settings', controller='admin/settings',

    #         path_prefix='/admin', name_prefix='admin_')

    @LoginRequired()

    def __before__(self):

        c.admin_user = session.get('admin_user')

        c.admin_username = session.get('admin_username')

        c.modules = sorted([(p.project_name, p.version)

        c.py_version = platform.python_version()

        c.platform = platform.platform()

        super(SettingsController, self).__before__()

    @HasPermissionAllDecorator('hg.admin')

    def index(self, format='html'):

        """GET /admin/settings: All items in the collection"""

        # url('admin_settings')

        defaults = RhodeCodeSetting.get_app_settings()

        defaults.update(self.get_hg_ui_settings())

        return htmlfill.render(

            render('admin/settings/settings.html'),

            defaults=defaults,

            encoding="UTF-8",

            force_defaults=False

        )

    @HasPermissionAllDecorator('hg.admin')

    def create(self):

        """POST /admin/settings: Create a new item"""

        # url('admin_settings')

    @HasPermissionAllDecorator('hg.admin')

    def new(self, format='html'):

        """GET /admin/settings/new: Form to create a new item"""

        # url('admin_new_setting')

    @HasPermissionAllDecorator('hg.admin')

    def update(self, setting_id):

        """PUT /admin/settings/setting_id: Update an existing item"""

        # Forms posted to this method should contain a hidden field:

        #    <input type="hidden" name="_method" value="PUT" />

        # Or using helpers:

        #    h.form(url('admin_setting', setting_id=ID),

        #           method='put')

        # url('admin_setting', setting_id=ID)

        if setting_id == 'mapping':

            rm_obsolete = request.POST.get('destroy', False)

            log.debug('Rescanning directories with destroy=%s' % rm_obsolete)

            initial = ScmModel().repo_scan()

            log.debug('invalidating all repositories')

            for repo_name in initial.keys():

                invalidate_cache('get_repo_cached_%s' % repo_name)

            added, removed = repo2db_mapper(initial, rm_obsolete)

            h.flash(_('Repositories successfully'

                      ' rescanned added: %s,removed: %s') % (added, removed),

                    category='success')

        if setting_id == 'whoosh':

            repo_location = self.get_hg_ui_settings()['paths_root_path']

            full_index = request.POST.get('full_index', False)

            run_task(tasks.whoosh_index, repo_location, full_index)

            h.flash(_('Whoosh reindex task scheduled'), category='success')

        if setting_id == 'global':

            application_form = ApplicationSettingsForm()()

            try:

                form_result = application_form.to_python(dict(request.POST))

                try:

                    hgsettings1 = RhodeCodeSetting.get_by_name('title')

                    hgsettings1.app_settings_value = \

                        form_result['rhodecode_title']

                    hgsettings2 = RhodeCodeSetting.get_by_name('realm')

                    hgsettings2.app_settings_value = \

                        form_result['rhodecode_realm']

                    hgsettings3 = RhodeCodeSetting.get_by_name('ga_code')

                    hgsettings3.app_settings_value = \

                        form_result['rhodecode_ga_code']

                    self.sa.add(hgsettings1)

                    self.sa.add(hgsettings2)

                    self.sa.add(hgsettings3)

                    self.sa.commit()

                    set_rhodecode_config(config)

                    h.flash(_('Updated application settings'),

                            category='success')

                except Exception:

                    log.error(traceback.format_exc())

                    h.flash(_('error occurred during updating '

                              'application settings'),

                            category='error')

                    self.sa.rollback()

            except formencode.Invalid, errors:

                return htmlfill.render(

                     render('admin/settings/settings.html'),

import rhodecode.lib.helpers as h

from rhodecode.lib.auth import LoginRequired, HasRepoPermissionAnyDecorator

from rhodecode.lib.base import BaseRepoController, render

from rhodecode.lib.helpers import RepoPage

from rhodecode.lib.compat import json

from rhodecode.lib.vcs.exceptions import RepositoryError, ChangesetDoesNotExistError

from rhodecode.model.db import Repository

log = logging.getLogger(__name__)

class ChangelogController(BaseRepoController):

    @LoginRequired()

    @HasRepoPermissionAnyDecorator('repository.read', 'repository.write',

                                   'repository.admin')

    def __before__(self):

        super(ChangelogController, self).__before__()

        c.affected_files_cut_off = 60

    def index(self):

        limit = 100

        default = 20

        if request.params.get('size'):

            try:

                int_size = int(request.params.get('size'))

            except ValueError:

                int_size = default

            int_size = int_size if int_size <= limit else limit

            c.size = int_size

            session['changelog_size'] = c.size

            session.save()

        else:

            c.size = int(session.get('changelog_size', default))

        p = int(request.params.get('page', 1))

        branch_name = request.params.get('branch', None)

        try:

            if branch_name:

                collection = [z for z in

                              c.rhodecode_repo.get_changesets(start=0,

                                                    branch_name=branch_name)]

                c.total_cs = len(collection)

            else:

                collection = c.rhodecode_repo

                c.total_cs = len(c.rhodecode_repo)

            c.pagination = RepoPage(collection, page=p, item_count=c.total_cs,

                                    items_per_page=c.size, branch=branch_name)

            collection = list(c.pagination)

            page_revisions = [x.raw_id for x in collection]

            c.comments = c.rhodecode_db_repo.comments(page_revisions)

            c.statuses = c.rhodecode_db_repo.statuses(page_revisions)

        except (RepositoryError, ChangesetDoesNotExistError, Exception), e:

            log.error(traceback.format_exc())

            h.flash(str(e), category='warning')

            return redirect(url('home'))

        self._graph(c.rhodecode_repo, collection, c.total_cs, c.size, p)

        c.branch_name = branch_name

        c.branch_filters = [('', _('All Branches'))] + \

            [(k, k) for k in c.rhodecode_repo.branches.keys()]

        return render('changelog/changelog.html')

    def changelog_details(self, cs):

        if request.environ.get('HTTP_X_PARTIAL_XHR'):

            c.cs = c.rhodecode_repo.get_changeset(cs)

            return render('changelog/changelog_details.html')

    def _graph(self, repo, collection, repo_size, size, p):

        """

        Generates a DAG graph for mercurial

        :param repo: repo instance

        :param size: number of commits to show

        :param p: page number

        """

        if not collection:

            c.jsdata = json.dumps([])

            return

        data = []

        revs = [x.revision for x in collection]

        if repo.alias == 'git':

            for _ in revs:

                vtx = [0, 1]

                edges = [[0, 0, 1]]

                data.append(['', vtx, edges])

        elif repo.alias == 'hg':

            dag = graphmod.dagwalker(repo._repo, revs)

            for (id, type, ctx, vtx, edges) in c.dag:

                if type != graphmod.CHANGESET:

                    continue

                data.append(['', vtx, edges])

        c.jsdata = json.dumps(data)

					</div>

				</div>

             </div>

             <div class="field">

                <div class="label label-checkbox">

                    <label>${_('Hooks')}:</label>

                </div>

                <div class="checkboxes">

					<div class="checkbox">

						${h.checkbox('hooks_changegroup_update','True')}

						<label for="hooks_changegroup_update">${_('Update repository after push (hg update)')}</label>

					</div>

					<div class="checkbox">

						${h.checkbox('hooks_changegroup_repo_size','True')}

						<label for="hooks_changegroup_repo_size">${_('Show repository size after push')}</label>

					</div>

                    <div class="checkbox">

                        ${h.checkbox('hooks_pretxnchangegroup_push_logger','True')}

                        <label for="hooks_pretxnchangegroup_push_logger">${_('Log user push commands')}</label>

                    </div>

                    <div class="checkbox">

                        ${h.checkbox('hooks_preoutgoing_pull_logger','True')}

                        <label for="hooks_preoutgoing_pull_logger">${_('Log user pull commands')}</label>

                    </div>

				</div>

                <div class="input" style="margin-top:10px">

                    ${h.link_to(_('advanced setup'),url('admin_edit_setting',setting_id='hooks'),class_="ui-btn")}

                </div>

             </div>

            <div class="field">

                <div class="label">

                    <label for="paths_root_path">${_('Repositories location')}:</label>

                </div>

                <div class="input">

                    ${h.text('paths_root_path',size=30,readonly="readonly")}

					<span id="path_unlock" class="tooltip"

						title="${h.tooltip(_('This a crucial application setting. If you are really sure you need to change this, you must restart application in order to make this setting take effect. Click this label to unlock.'))}">

		                ${_('unlock')}</span>

                </div>

            </div>

            <div class="buttons">

                ${h.submit('save',_('Save settings'),class_="ui-button")}

                ${h.reset('reset',_('Reset'),class_="ui-button")}

           </div>

        </div>

    </div>

    ${h.end_form()}

    <script type="text/javascript">

        YAHOO.util.Event.onDOMReady(function(){

            YAHOO.util.Event.addListener('path_unlock','click',function(){

                YAHOO.util.Dom.get('paths_root_path').removeAttribute('readonly');

            });

        });

    </script>

    <h3>${_('Test Email')}</h3>

    ${h.form(url('admin_setting', setting_id='email'),method='put')}

    <div class="form">

        <!-- fields -->

        <div class="fields">

            <div class="field">

                <div class="label">

                    <label for="test_email">${_('Email to')}:</label>

                </div>

                <div class="input">

                    ${h.text('test_email',size=30)}

                </div>

            </div>

            <div class="buttons">

            ${h.submit('send',_('Send'),class_="ui-button")}

            </div>

        </div>

    </div>

    ${h.end_form()}

    <h3>${_('System Info and Packages')}</h3>

    <div class="form">

    <div>

        <h5 id="expand_modules" style="cursor: pointer">&darr; ${_('show')} &darr;</h5>

    </div>

      <div id="expand_modules_table"  style="display:none">

      <h5>Python - ${c.py_version}</h5>

      <h5>System - ${c.platform}</h5>

      <table class="table" style="margin:0px 0px 0px 20px">

          <colgroup>

              <col style="width:220px">

          </colgroup>

          <tbody>

              %for key, value in c.modules:

                  <tr>

                      <td>${value}</td>

                  </tr>

              %endfor

          </tbody>

      </table>

      </div>

    </div>

    <script type="text/javascript">

    YUE.on('expand_modules','click',function(e){

    	YUD.setStyle('expand_modules_table','display','');

    	YUD.setStyle('expand_modules','display','none');

    })

    </script>

</div>

</%def>

<dl>

	<dt style="padding-top:10px;font-size:16px">${_('History')}</dt>

	<dd>

		<div>

		${h.form(h.url('files_diff_home',repo_name=c.repo_name,f_path=c.f_path),method='get')}

		${h.hidden('diff2',c.file.changeset.raw_id)}

		${h.select('diff1',c.file.changeset.raw_id,c.file_history)}

		${h.submit('diff','diff to revision',class_="ui-btn")}

		${h.submit('show_rev','show at revision',class_="ui-btn")}

		${h.end_form()}

		</div>

	</dd>

</dl>

<div id="body" class="codeblock">

	<div class="code-header">

        <div class="stats">

            <div class="left img"><img src="${h.url('/images/icons/file.png')}"/></div>

            <div class="left item"><pre>${h.format_byte_size(c.file.size,binary=True)}</pre></div>

            <div class="left item last"><pre>${c.file.mimetype}</pre></div>

            <div class="buttons">

              %if c.annotate:

                ${h.link_to(_('show source'),    h.url('files_home',         repo_name=c.repo_name,revision=c.file.changeset.raw_id,f_path=c.f_path),class_="ui-btn")}

              %else:

                ${h.link_to(_('show annotation'),h.url('files_annotate_home',repo_name=c.repo_name,revision=c.file.changeset.raw_id,f_path=c.f_path),class_="ui-btn")}

              %endif

              ${h.link_to(_('show as raw'),h.url('files_raw_home',repo_name=c.repo_name,revision=c.file.changeset.raw_id,f_path=c.f_path),class_="ui-btn")}

              ${h.link_to(_('download as raw'),h.url('files_rawfile_home',repo_name=c.repo_name,revision=c.file.changeset.raw_id,f_path=c.f_path),class_="ui-btn")}

              % if h.HasRepoPermissionAny('repository.write','repository.admin')(c.repo_name):

               % if not c.file.is_binary:

                ${h.link_to(_('edit'),h.url('files_edit_home',repo_name=c.repo_name,revision=c.file.changeset.raw_id,f_path=c.f_path),class_="ui-btn")}

               % endif

              % endif

            </div>

        </div>

        <div class="author">

            <div class="gravatar">

                <img alt="gravatar" src="${h.gravatar_url(h.email(c.file.changeset.author),16)}"/>

            </div>

            <div title="${c.file.changeset.author}" class="user">${h.person(c.file.changeset.author)}</div>

        </div>

		<div class="commit">${h.urlify_commit(c.file.changeset.message,c.repo_name)}</div>

	</div>

	<div class="code-body">

	   %if c.file.is_binary:

	       ${_('Binary file (%s)') % c.file.mimetype}

	   %else:

		% if c.file.size < c.cut_off_limit:

            %if c.annotate:

              ${h.pygmentize_annotation(c.repo_name,c.file,linenos=True,anchorlinenos=True,lineanchors='L',cssclass="code-highlight")}

            %else:

			  ${h.pygmentize(c.file,linenos=True,anchorlinenos=True,lineanchors='L',cssclass="code-highlight")}

            %endif

		%else:

			${_('File is too big to display')} ${h.link_to(_('show as raw'),

			h.url('files_raw_home',repo_name=c.repo_name,revision=c.file.changeset.raw_id,f_path=c.f_path))}

		%endif

     %endif

	</div>

</div>

<script type="text/javascript">

YUE.onDOMReady(function(){

    function highlight_lines(lines){

        for(pos in lines){

          YUD.setStyle('L'+lines[pos],'background-color','#FFFFBE');

        }

    }

    page_highlights = location.href.substring(location.href.indexOf('#')+1).split('L');

    if (page_highlights.length == 2){

       highlight_ranges  = page_highlights[1].split(",");

       var h_lines = [];

       for (pos in highlight_ranges){

            var _range = highlight_ranges[pos].split('-');

            if(_range.length == 2){

                var start = parseInt(_range[0]);

                var end = parseInt(_range[1]);

                if (start < end){

                    for(var i=start;i<=end;i++){

                        h_lines.push(i);

                    }

                }

            }

            else{

                h_lines.push(parseInt(highlight_ranges[pos]));

            }

      }

    highlight_lines(h_lines);

    //remember original location

    var old_hash  = location.href.substring(location.href.indexOf('#'));

    // this makes a jump to anchor moved by 3 posstions for padding

    window.location.hash = '#L'+Math.max(parseInt(h_lines[0])-3,1);

    //sets old anchor

    window.location.hash = old_hash;

    }

    YUE.on('show_rev','click',function(e){

    	YUE.preventDefault(e);

        var cs = YUD.get('diff1').value;

        %if c.annotate:

          var url = "${h.url('files_annotate_home',repo_name=c.repo_name,revision='__CS__',f_path=c.f_path)}".replace('__CS__',cs);

        %else:

          var url = "${h.url('files_home',repo_name=c.repo_name,revision='__CS__',f_path=c.f_path)}".replace('__CS__',cs);

        %endif

        window.location = url;

    });

    YUE.on('hlcode','mouseup',getSelectionLink("${_('Selection link')}"))

   });

</script>

# -*- coding: utf-8 -*-

"""

    rhodecode.tests.test_crawer

    ~~~~~~~~~~~~~~~~~~~~~~~~~~~

    Test for crawling a project for memory usage

    This should be runned just as regular script together

    with a watch script that will show memory usage.

    watch -n1 ./rhodecode/tests/mem_watch

    :created_on: Apr 21, 2010

    :author: marcink

    :copyright: (C) 2010-2012 Marcin Kuzminski <marcin@python-works.com>

    :license: GPLv3, see COPYING for more details.

"""

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import cookielib

import urllib

import urllib2

import time

import os

import sys

from os.path import join as jn

from os.path import dirname as dn

__here__ = os.path.abspath(__file__)

__root__ = dn(dn(dn(__here__)))

sys.path.append(__root__)

from rhodecode.lib import vcs

PROJECT_PATH = jn('/', 'home', 'marcink', 'hg_repos')

PROJECTS = [

    'linux-magx-pbranch',

    'CPython',

    'rhodecode_tip',

]

cj = cookielib.FileCookieJar('/tmp/rc_test_cookie.txt')

o = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))

o.addheaders = [

    ('User-agent', 'rhodecode-crawler'),

    ('Accept-Language', 'en - us, en;q = 0.5')

]

urllib2.install_opener(o)

def test_changelog_walk(proj, pages=100):

    total_time = 0

    for i in range(1, pages):

        page = '/'.join((proj, 'changelog',))

        full_uri = (BASE_URI % page) + '?' + urllib.urlencode({'page':i})

        s = time.time()

        f = o.open(full_uri)

        size = len(f.read())

        e = time.time() - s

        total_time += e

        print 'visited %s size:%s req:%s ms' % (full_uri, size, e)

    print 'total_time', total_time

    print 'average on req', total_time / float(pages)

def test_changeset_walk(proj, limit=None):

    print 'processing', jn(PROJECT_PATH, proj)

    total_time = 0

    cnt = 0

    for i in repo:

        cnt += 1

        raw_cs = '/'.join((proj, 'changeset', i.raw_id))

        if limit and limit == cnt:

            break

        full_uri = (BASE_URI % raw_cs)

        print '%s visiting %s\%s' % (cnt, full_uri, i)

        s = time.time()

        f = o.open(full_uri)

        size = len(f.read())

        e = time.time() - s

        total_time += e

        print '%s visited %s\%s size:%s req:%s ms' % (cnt, full_uri, i, size, e)

    print 'total_time', total_time

    print 'average on req', total_time / float(cnt)

def test_files_walk(proj, limit=100):

    print 'processing', jn(PROJECT_PATH, proj)

    total_time = 0

    paths_ = OrderedSet([''])

    try:

        tip = repo.get_changeset('tip')

        for topnode, dirs, files in tip.walk('/'):

            for dir in dirs:

                paths_.add(dir.path)

                for f in dir:

                    paths_.add(f.path)

            for f in files:

                paths_.add(f.path)

    except RepositoryError, e:

        pass

    cnt = 0

    for f in paths_:

        cnt += 1

        if limit and limit == cnt:

            break

        file_path = '/'.join((proj, 'files', 'tip', f))

        full_uri = (BASE_URI % file_path)

        print '%s visiting %s' % (cnt, full_uri)

        s = time.time()

        f = o.open(full_uri)

        size = len(f.read())

        e = time.time() - s

        total_time += e

        print '%s visited OK size:%s req:%s ms' % (cnt, size, e)

    print 'total_time', total_time

    print 'average on req', total_time / float(cnt)

if __name__ == '__main__':