Files
@ f734d107296e
Branch filter:
Location: kallithea/docs/upgrade.rst - annotation
f734d107296e
8.0 KiB
text/prs.fallenstein.rst
auth: for default permissions, use existing explicit query result values instead of following dot references in ORM result objects
There has been reports of spurious crashes on resolving references like
.repository from Permissions:
File ".../kallithea/lib/auth.py", line 678, in __wrapper
if self.check_permissions(user):
File ".../kallithea/lib/auth.py", line 718, in check_permissions
return user.has_repository_permission_level(repo_name, self.required_perm)
File ".../kallithea/lib/auth.py", line 450, in has_repository_permission_level
actual_perm = self.permissions['repositories'].get(repo_name)
File ".../kallithea/lib/vcs/utils/lazy.py", line 41, in __get__
value = self._func(obj)
File ".../kallithea/lib/auth.py", line 442, in permissions
return self.__get_perms(user=self, cache=False)
File ".../kallithea/lib/auth.py", line 498, in __get_perms
return compute(user_id, user_is_admin)
File ".../kallithea/lib/auth.py", line 190, in _cached_perms_data
r_k = perm.UserRepoToPerm.repository.repo_name
File ".../sqlalchemy/orm/attributes.py", line 285, in __get__
return self.impl.get(instance_state(instance), dict_)
File ".../sqlalchemy/orm/attributes.py", line 721, in get
value = self.callable_(state, passive)
File ".../sqlalchemy/orm/strategies.py", line 710, in _load_for_state
% (orm_util.state_str(state), self.key)
sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <UserRepoToPerm at ...> is not bound to a Session; lazy load operation of attribute 'repository' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)
Permissions are cached between requests: SA result records are stored in in
beaker.cache.sql_cache_short and resued in following requests after the initial
session as been removed. References in Permission objects would usually give
lazy lookup ... but not outside the original session, where we would get an
error like this.
Permissions are indeed implemented/used incorrectly. That might explain a part
of the problem. Even if not fully explaining or fixing this problem, it is
still worth fixing:
Permissions are fetched from the database using Session().query with multiple
class/table names (joined together in way that happens to match the references
specified in the table definitions) - including Repository. The results are
thus "structs" with selected objects. If repositories always were retrieved
using this selected repository, everything would be fine. In some places, this
was what we did.
But in some places, the code happened to do what was more intuitive: just use
.repository and rely on "lazy" resolving. SA was not aware that this one
already was present in the result struct, and would try to fetch it again. Best
case, that could be inefficient. Worst case, it would fail as we see here.
Fix this by only querying from one table but use the "joinedload" option to
also fetch other referenced tables in the same select. (This might
inefficiently return the main record multiple times ... but that was already
the case with the previous approach.)
This change is thus doing multiple things with circular dependencies that can't
be split up in minor parts without taking detours:
The existing repository join like:
.join((Repository, UserGroupRepoToPerm.repository_id == Repository.repo_id))
is thus replaced by:
.options(joinedload(UserGroupRepoToPerm.repository))
Since we only are doing Session.query() on one table, the results will be of
that type instead of "structs" with multiple objects. If only querying for
UserRepoToPerm this means:
- perm.UserRepoToPerm.repository becomes perm.repository
- perm.Permission.permission_name looked at the explicitly queried Permission
in the result struct - instead it should look in the the dereferenced
repository as perm.permission.permission_name
There has been reports of spurious crashes on resolving references like
.repository from Permissions:
File ".../kallithea/lib/auth.py", line 678, in __wrapper
if self.check_permissions(user):
File ".../kallithea/lib/auth.py", line 718, in check_permissions
return user.has_repository_permission_level(repo_name, self.required_perm)
File ".../kallithea/lib/auth.py", line 450, in has_repository_permission_level
actual_perm = self.permissions['repositories'].get(repo_name)
File ".../kallithea/lib/vcs/utils/lazy.py", line 41, in __get__
value = self._func(obj)
File ".../kallithea/lib/auth.py", line 442, in permissions
return self.__get_perms(user=self, cache=False)
File ".../kallithea/lib/auth.py", line 498, in __get_perms
return compute(user_id, user_is_admin)
File ".../kallithea/lib/auth.py", line 190, in _cached_perms_data
r_k = perm.UserRepoToPerm.repository.repo_name
File ".../sqlalchemy/orm/attributes.py", line 285, in __get__
return self.impl.get(instance_state(instance), dict_)
File ".../sqlalchemy/orm/attributes.py", line 721, in get
value = self.callable_(state, passive)
File ".../sqlalchemy/orm/strategies.py", line 710, in _load_for_state
% (orm_util.state_str(state), self.key)
sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <UserRepoToPerm at ...> is not bound to a Session; lazy load operation of attribute 'repository' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)
Permissions are cached between requests: SA result records are stored in in
beaker.cache.sql_cache_short and resued in following requests after the initial
session as been removed. References in Permission objects would usually give
lazy lookup ... but not outside the original session, where we would get an
error like this.
Permissions are indeed implemented/used incorrectly. That might explain a part
of the problem. Even if not fully explaining or fixing this problem, it is
still worth fixing:
Permissions are fetched from the database using Session().query with multiple
class/table names (joined together in way that happens to match the references
specified in the table definitions) - including Repository. The results are
thus "structs" with selected objects. If repositories always were retrieved
using this selected repository, everything would be fine. In some places, this
was what we did.
But in some places, the code happened to do what was more intuitive: just use
.repository and rely on "lazy" resolving. SA was not aware that this one
already was present in the result struct, and would try to fetch it again. Best
case, that could be inefficient. Worst case, it would fail as we see here.
Fix this by only querying from one table but use the "joinedload" option to
also fetch other referenced tables in the same select. (This might
inefficiently return the main record multiple times ... but that was already
the case with the previous approach.)
This change is thus doing multiple things with circular dependencies that can't
be split up in minor parts without taking detours:
The existing repository join like:
.join((Repository, UserGroupRepoToPerm.repository_id == Repository.repo_id))
is thus replaced by:
.options(joinedload(UserGroupRepoToPerm.repository))
Since we only are doing Session.query() on one table, the results will be of
that type instead of "structs" with multiple objects. If only querying for
UserRepoToPerm this means:
- perm.UserRepoToPerm.repository becomes perm.repository
- perm.Permission.permission_name looked at the explicitly queried Permission
in the result struct - instead it should look in the the dereferenced
repository as perm.permission.permission_name
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 | 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 0b6d2ca7175f 0b6d2ca7175f 0b6d2ca7175f 0b6d2ca7175f 0b6d2ca7175f 0b6d2ca7175f 0b6d2ca7175f 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 f973b866fffc 874ed3e58579 874ed3e58579 8075ec3d0233 8075ec3d0233 de92f48c1375 de92f48c1375 de92f48c1375 de92f48c1375 de92f48c1375 8075ec3d0233 de92f48c1375 de92f48c1375 de92f48c1375 de92f48c1375 de92f48c1375 de92f48c1375 de92f48c1375 de92f48c1375 de92f48c1375 de92f48c1375 de92f48c1375 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 de92f48c1375 de92f48c1375 de92f48c1375 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 634c2fa427e8 634c2fa427e8 8075ec3d0233 8075ec3d0233 634c2fa427e8 634c2fa427e8 634c2fa427e8 634c2fa427e8 29e9cb56f26f 6ef837acb0d2 14e8dcffd279 14e8dcffd279 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 213085032127 8075ec3d0233 7784a1212471 213085032127 874ed3e58579 874ed3e58579 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 9fd64dd2617d 9fd64dd2617d 8075ec3d0233 8075ec3d0233 8075ec3d0233 8075ec3d0233 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 14e8dcffd279 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 14e8dcffd279 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 14e8dcffd279 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 14e8dcffd279 14e8dcffd279 14e8dcffd279 14e8dcffd279 14e8dcffd279 14e8dcffd279 14e8dcffd279 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 9fd64dd2617d 8075ec3d0233 8075ec3d0233 a2abde4062ec a2abde4062ec a2abde4062ec a2abde4062ec a2abde4062ec a2abde4062ec a2abde4062ec a2abde4062ec a2abde4062ec a2abde4062ec b3644eeee445 b3644eeee445 b3644eeee445 b3644eeee445 b3644eeee445 b3644eeee445 b3644eeee445 b3644eeee445 b3644eeee445 b3644eeee445 b3644eeee445 b3644eeee445 b3644eeee445 874ed3e58579 874ed3e58579 874ed3e58579 874ed3e58579 874ed3e58579 b3644eeee445 b3644eeee445 b3644eeee445 b3644eeee445 b3644eeee445 b3644eeee445 b3644eeee445 b3644eeee445 b3644eeee445 b3644eeee445 415cc651bd83 415cc651bd83 415cc651bd83 415cc651bd83 415cc651bd83 415cc651bd83 415cc651bd83 415cc651bd83 415cc651bd83 415cc651bd83 415cc651bd83 415cc651bd83 415cc651bd83 415cc651bd83 | .. _upgrade:
===================
Upgrading Kallithea
===================
This describes the process for upgrading Kallithea, independently of the
Kallithea installation method.
.. note::
If you are upgrading from a RhodeCode installation, you must first
install Kallithea 0.3.2 and follow the instructions in the 0.3.2
README to perform a one-time conversion of the database from
RhodeCode to Kallithea, before upgrading to the latest version
of Kallithea.
1. Stop the Kallithea web application
-------------------------------------
This step depends entirely on the web server software used to serve
Kallithea, but in any case, Kallithea should not be running during
the upgrade.
.. note::
If you're using Celery, make sure you stop all instances during the
upgrade.
2. Create a backup of both database and configuration
-----------------------------------------------------
You are of course strongly recommended to make backups regularly, but it
is *especially* important to make a full database and configuration
backup before performing a Kallithea upgrade.
Back up your configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^
Make a copy of your Kallithea configuration (``.ini``) file.
If you are using :ref:`rcextensions <customization>`, you should also
make a copy of the entire ``rcextensions`` directory.
Back up your database
^^^^^^^^^^^^^^^^^^^^^
If using SQLite, simply make a copy of the Kallithea database (``.db``)
file.
If using PostgreSQL, please consult the documentation for the ``pg_dump``
utility.
If using MySQL, please consult the documentation for the ``mysqldump``
utility.
Look for ``sqlalchemy.url`` in your configuration file to determine
database type, settings, location, etc. If you were running Kallithea 0.3.x or
older, this was ``sqlalchemy.db1.url``.
3. Activate or recreate the Kallithea virtual environment (if any)
------------------------------------------------------------------
.. note::
If you did not install Kallithea in a virtual environment, skip this step.
For major upgrades, e.g. from 0.3.x to 0.4.x, it is recommended to create a new
virtual environment, rather than reusing the old. For minor upgrades, e.g.
within the 0.4.x range, this is not really necessary (but equally fine).
To create a new virtual environment, please refer to the appropriate
installation page for details. After creating and activating the new virtual
environment, proceed with the rest of the upgrade process starting from the next
section.
To reuse the same virtual environment, first activate it, then verify that you
are using the correct environment by running::
pip freeze
This will list all packages installed in the current environment. If
Kallithea isn't listed, deactivate the environment and then activate the correct
one, or recreate a new environment. See the appropriate installation page for
details.
4. Install new version of Kallithea
-----------------------------------
Please refer to the instructions for the installation method you
originally used to install Kallithea.
If you originally installed using pip, it is as simple as::
pip install --upgrade kallithea
If you originally installed from version control, assuming you did not make
private changes (in which case you should adapt the instructions accordingly)::
cd my-kallithea-clone
hg parent # make a note of the original revision
hg pull
hg update
hg parent # make a note of the new revision
pip install --upgrade -e .
.. _upgrade_config:
5. Upgrade your configuration
-----------------------------
Run the following command to create a new configuration (``.ini``) file::
kallithea-cli config-create new.ini
Then compare it with your old config file and copy over the required
configuration values from the old to the new file.
.. note::
Please always make sure your ``.ini`` files are up to date. Errors
can often be caused by missing parameters added in new versions.
.. _upgrade_db:
6. Upgrade your database
------------------------
.. note::
If you are *downgrading* Kallithea, you should perform the database
migration step *before* installing the older version. (That is,
always perform migrations using the most recent of the two versions
you're migrating between.)
First, run the following command to see your current database version::
alembic -c new.ini current
Typical output will be something like "9358dc3d6828 (head)", which is
the current Alembic database "revision ID". Write down the entire output
for troubleshooting purposes.
The output will be empty if you're upgrading from Kallithea 0.3.x or
older. That's expected. If you get an error that the config file was not
found or has no ``[alembic]`` section, see the next section.
Next, if you are performing an *upgrade*: Run the following command to
upgrade your database to the current Kallithea version::
alembic -c new.ini upgrade head
If you are performing a *downgrade*: Run the following command to
downgrade your database to the given version::
alembic -c new.ini downgrade 0.4
Alembic will show the necessary migrations (if any) as it executes them.
If no "ERROR" is displayed, the command was successful.
Should an error occur, the database may be "stranded" half-way
through the migration, and you should restore it from backup.
Enabling old Kallithea config files for Alembic use
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Kallithea configuration files created before the introduction of Alembic
(i.e. predating Kallithea 0.4) need to be updated for use with Alembic.
Without this, Alembic will fail with an error like this::
FAILED: No config file 'my.ini' found, or file has no '[alembic]' section
.. note::
If you followed this upgrade guide correctly, you will have created a
new configuration file in section :ref:`Upgrading your configuration
<upgrade_config>`. When calling Alembic, make
sure to use this new config file. In this case, you should not get any
errors and the below manual steps should not be needed.
If Alembic complains specifically about a missing ``alembic.ini``, it is
likely because you did not specify a config file using the ``-c`` option.
On the other hand, if the mentioned config file actually exists, you
need to append the following lines to it::
[alembic]
script_location = kallithea:alembic
Your config file should now work with Alembic.
7. Prepare the front-end
------------------------
Starting with Kallithea 0.4, external front-end dependencies are no longer
shipped but need to be downloaded and/or generated at installation time. Run the
following command::
kallithea-cli front-end-build
8. Rebuild the Whoosh full-text index
-------------------------------------
It is recommended that you rebuild the Whoosh index after upgrading since
new Whoosh versions can introduce incompatible index changes.
9. Start the Kallithea web application
--------------------------------------
This step once again depends entirely on the web server software used to
serve Kallithea.
If you were running Kallithea 0.3.x or older and were using ``paster serve
my.ini`` before, then the corresponding command in Kallithea 0.4 and later is::
gearbox serve -c new.ini
Before starting the new version of Kallithea, you may find it helpful to
clear out your log file so that new errors are readily apparent.
.. note::
If you're using Celery, make sure you restart all instances of it after
upgrade.
10. Update Git repository hooks
-------------------------------
It is possible that an upgrade involves changes to the Git hooks installed by
Kallithea. As these hooks are created inside the repositories on the server
filesystem, they are not updated automatically when upgrading Kallithea itself.
To update the hooks of your Git repositories:
* Go to *Admin > Settings > Remap and Rescan*
* Select the checkbox *Install Git hooks*
* Click the button *Rescan repositories*
.. note::
Kallithea does not use hooks on Mercurial repositories. This step is thus
not necessary if you only have Mercurial repositories.
|