Changeset - 716e53c085ff
Parent rev.
Child rev.
[Not reviewed]
stable
0 6 0
Mads Kiilerich - 9 years ago 2016-09-06 00:51:18
madski@unity3d.com
Grafted from: d6942b2b421c
config: clarify that we only recommend and support single threaded operation

Sad, but true. Especially because we reuse Repository instances between
threads.
6 files changed with 35 insertions and 18 deletions:
development.ini
3
3
docs/setup.rst
2
3
docs/usage/performance.rst
21
3
kallithea/bin/template.ini.mako
3
3
kallithea/config/deployment.ini_tmpl
3
3
kallithea/tests/test.ini
3
3
0 comments (0 inline, 0 general)
development.ini
➞
Show inline comments
 
@@ -71,16 +71,16 @@ pdebug = false
 
## PASTE ##
 
#use = egg:Paste#http
 
## nr of worker threads to spawn
 
#threadpool_workers = 5
 
#threadpool_workers = 1
 
## max request before thread respawn
 
#threadpool_max_requests = 10
 
#threadpool_max_requests = 100
 
## option to use threads of process
 
#use_threadpool = true
 

			




	
	
	
		
 
## WAITRESS ##

			




	
	
	
		
 
use = egg:waitress#main

			




	
	
	
		
 
## number of worker threads

			




	
	
	
		
 
threads = 5

			




	
	
	
		
 
threads = 1

			




	
	
	
		
 
## MAX BODY SIZE 100GB

			




	
	
	
		
 
max_request_body_size = 107374182400

			




	
	
	
		
 
## use poll instead of select, fixes fd limits, may not work on old

			




        

    


    
    

    

        

                

                    docs/setup.rst
                

                

                  
                      
                        

                      ➞

                      
                        
                  

                  
                      
                  
                      
                  
                      
                  
                      
                  
                  
                

                

                    Show inline comments
                    
                

        

        

            



	
	
		
 
@@ -735,8 +735,7 @@ Here is a sample excerpt from an Apache 

			




	
	
	
		
 

			




	
	
	
		
 
.. code-block:: apache

			




	
	
	
		
 

			




	
	
	
		
 
    WSGIDaemonProcess kallithea \

			




	
	
	
		
 
        threads=4 \

			




	
	
	
		
 
    WSGIDaemonProcess kallithea processes=5 threads=1 maximum-requests=100 \

			




	
	
	
		
 
        python-home=/srv/kallithea/venv

			




	
	
	
		
 
    WSGIProcessGroup kallithea

			




	
	
	
		
 
    WSGIScriptAlias / /srv/kallithea/dispatch.wsgi

			




	
	
		
 
@@ -746,7 +745,7 @@ Or if using a dispatcher WSGI script wit

			




	
	
	
		
 

			




	
	
	
		
 
.. code-block:: apache

			




	
	
	
		
 

			




	
	
	
		
 
    WSGIDaemonProcess kallithea threads=4

			




	
	
	
		
 
    WSGIDaemonProcess kallithea processes=5 threads=1 maximum-requests=100

			




	
	
	
		
 
    WSGIProcessGroup kallithea

			




	
	
	
		
 
    WSGIScriptAlias / /srv/kallithea/dispatch.wsgi

			




	
	
	
		
 
    WSGIPassAuthorization On

			




        

    


    
    

    

        

                

                    docs/usage/performance.rst
                

                

                  
                      
                        

                      ➞

                      
                        
                  

                  
                      
                  
                      
                  
                      
                  
                      
                  
                  
                

                

                    Show inline comments
                    
                

        

        

            



	
	
		
 
@@ -35,8 +35,26 @@ Follow these few steps to improve perfor

			




	
	
	
		
 

			




	
	
	
		
 
    Scaling horizontally can give huge performance benefits when dealing with

			




	
	
	
		
 
    large amounts of traffic (many users, CI servers, etc.). Kallithea can be

			




	
	
	
		
 
    scaled horizontally on one (recommended) or multiple machines. In order

			




	
	
	
		
 
    to scale horizontally you need to do the following:

			




	
	
	
		
 
    scaled horizontally on one (recommended) or multiple machines.

			




	
	
	
		
 

			




	
	
	
		
 
    It is generally possible to run WSGI applications multithreaded, so that

			




	
	
	
		
 
    several HTTP requests are served from the same Python process at once. That

			




	
	
	
		
 
    can in principle give better utilization of internal caches and less

			




	
	
	
		
 
    process overhead.

			




	
	
	
		
 

			




	
	
	
		
 
    One danger of running multithreaded is that program execution becomes much

			




	
	
	
		
 
    more complex; programs must be written to consider all combinations of

			




	
	
	
		
 
    events and problems might depend on timing and be impossible to reproduce.

			




	
	
	
		
 

			




	
	
	
		
 
    Kallithea can't promise to be thread-safe, just like the embedded Mercurial

			




	
	
	
		
 
    backend doesn't make any strong promises when used as Kallithea uses it.

			




	
	
	
		
 
    Instead, we recommend scaling by using multiple server processes.

			




	
	
	
		
 

			




	
	
	
		
 
    Web servers with multiple worker processes (such as ``mod_wsgi`` with the

			




	
	
	
		
 
    ``WSGIDaemonProcess`` ``processes`` parameter) will work out of the box.

			




	
	
	
		
 

			




	
	
	
		
 
    In order to scale horizontally on multiple machines, you need to do the

			




	
	
	
		
 
    following:

			




	
	
	
		
 

			




	
	
	
		
 
    - Each instance needs its own .ini file and unique ``instance_id`` set.

			




	
	
	
		
 
    - Each instance's ``data`` storage needs to be configured to be stored on a

			




	
	
		
 
@@ -45,7 +63,7 @@ Follow these few steps to improve perfor

			




	
	
	
		
 
      task locking (so it is safe across multiple instances). Set the

			




	
	
	
		
 
      ``cache_dir``, ``index_dir``, ``beaker.cache.data_dir``, ``beaker.cache.lock_dir``

			




	
	
	
		
 
      variables in each .ini file to a shared location across Kallithea instances

			




	
	
	
		
 
    - If celery is used each instance should run a separate Celery instance, but

			




	
	
	
		
 
    - If using several Celery instances,

			




	
	
	
		
 
      the message broker should be common to all of them (e.g.,  one

			




	
	
	
		
 
      shared RabbitMQ server)

			




	
	
	
		
 
    - Load balance using round robin or IP hash, recommended is writing LB rules

			




        

    


    
    

    

        

                

                    kallithea/bin/template.ini.mako
                

                

                  
                      
                        

                      ➞

                      
                        
                  

                  
                      
                  
                      
                  
                      
                  
                      
                  
                  
                

                

                    Show inline comments
                    
                

        

        

            



	
	
		
 
@@ -66,9 +66,9 @@ pdebug = false

			




	
	
	
		
 
<%text>## PASTE ##</%text>

			




	
	
	
		
 
use = egg:Paste#http

			




	
	
	
		
 
<%text>## nr of worker threads to spawn</%text>

			




	
	
	
		
 
threadpool_workers = 5

			




	
	
	
		
 
threadpool_workers = 1

			




	
	
	
		
 
<%text>## max request before thread respawn</%text>

			




	
	
	
		
 
threadpool_max_requests = 10

			




	
	
	
		
 
threadpool_max_requests = 100

			




	
	
	
		
 
<%text>## option to use threads of process</%text>

			




	
	
	
		
 
use_threadpool = true

			




	
	
	
		
 

			




	
	
		
 
@@ -76,7 +76,7 @@ use_threadpool = true

			




	
	
	
		
 
<%text>## WAITRESS ##</%text>

			




	
	
	
		
 
use = egg:waitress#main

			




	
	
	
		
 
<%text>## number of worker threads</%text>

			




	
	
	
		
 
threads = 5

			




	
	
	
		
 
threads = 1

			




	
	
	
		
 
<%text>## MAX BODY SIZE 100GB</%text>

			




	
	
	
		
 
max_request_body_size = 107374182400

			




	
	
	
		
 
<%text>## use poll instead of select, fixes fd limits, may not work on old</%text>

			




        

    


    
    

    

        

                

                    kallithea/config/deployment.ini_tmpl
                

                

                  
                      
                        

                      ➞

                      
                        
                  

                  
                      
                  
                      
                  
                      
                  
                      
                  
                  
                

                

                    Show inline comments
                    
                

        

        

            



	
	
		
 
@@ -66,16 +66,16 @@ pdebug = false

			




	
	
	
		
 
## PASTE ##

			




	
	
	
		
 
#use = egg:Paste#http

			




	
	
	
		
 
## nr of worker threads to spawn

			




	
	
	
		
 
#threadpool_workers = 5

			




	
	
	
		
 
#threadpool_workers = 1

			




	
	
	
		
 
## max request before thread respawn

			




	
	
	
		
 
#threadpool_max_requests = 10

			




	
	
	
		
 
#threadpool_max_requests = 100

			




	
	
	
		
 
## option to use threads of process

			




	
	
	
		
 
#use_threadpool = true

			




	
	
	
		
 

			




	
	
	
		
 
## WAITRESS ##

			




	
	
	
		
 
use = egg:waitress#main

			




	
	
	
		
 
## number of worker threads

			




	
	
	
		
 
threads = 5

			




	
	
	
		
 
threads = 1

			




	
	
	
		
 
## MAX BODY SIZE 100GB

			




	
	
	
		
 
max_request_body_size = 107374182400

			




	
	
	
		
 
## use poll instead of select, fixes fd limits, may not work on old

			




        

    


    
    

    

        

                

                    kallithea/tests/test.ini
                

                

                  
                      
                        

                      ➞

                      
                        
                  

                  
                      
                  
                      
                  
                      
                  
                      
                  
                  
                

                

                    Show inline comments
                    
                

        

        

            



	
	
		
 
@@ -70,16 +70,16 @@ pdebug = false

			




	
	
	
		
 
## PASTE ##

			




	
	
	
		
 
#use = egg:Paste#http

			




	
	
	
		
 
## nr of worker threads to spawn

			




	
	
	
		
 
#threadpool_workers = 5

			




	
	
	
		
 
#threadpool_workers = 1

			




	
	
	
		
 
## max request before thread respawn

			




	
	
	
		
 
#threadpool_max_requests = 10

			




	
	
	
		
 
#threadpool_max_requests = 100

			




	
	
	
		
 
## option to use threads of process

			




	
	
	
		
 
#use_threadpool = true

			




	
	
	
		
 

			




	
	
	
		
 
## WAITRESS ##

			




	
	
	
		
 
use = egg:waitress#main

			




	
	
	
		
 
## number of worker threads

			




	
	
	
		
 
threads = 5

			




	
	
	
		
 
threads = 1

			




	
	
	
		
 
## MAX BODY SIZE 100GB

			




	
	
	
		
 
max_request_body_size = 107374182400

			




	
	
	
		
 
## use poll instead of select, fixes fd limits, may not work on old

			




        

    



    


    



    



      

      





    
    0 comments (0 inline, 0 general)
    





    


  

  







  


    




    








    
        
    
    
        This site is powered by
            Kallithea 0.7.0,
        which is
        © 2010–2025 by various authors & licensed under GPLv3.