kallithea Changeset - 7e67883ee300

Changeset - 7e67883ee300

Parent rev.

Child rev.

[Not reviewed]

default

0 1 0

domruf - 10 years ago 2016-03-03 18:44:08
dominikruf@gmail.com

vcs: fix processing of git commands that return output on stderr

The subprocessio module used for interfacing with Git is using threading. It
might not be fully deterministic.

The subprocessio module also had the "feature" that it stopped reading and
reported failure once it found output on stderr - even if the command completed
(or would complete) with success.

Sometimes (like https://drone.io/bitbucket.org/domruf/kallithea/24 ) tests on
linux could fail on totally normal informational output like:

Couldn't run git command (['git', '-c', 'core.quotepath=false', 'checkout', 'foobranch']).
Original error was: Subprocess exited due to an error:
Switched to branch 'foobranch'

On Windows it would fail even more often.

To fix that, ignore stderr while processing output. There is a risk that it in
some cases can make the process block on stderr and thus never finish ... but
there is no reports of that yet.

1 file changed with 1 insertions and 1 deletions:

kallithea/lib/vcs/subprocessio.py

0 comments (0 inline, 0 general)

kallithea/lib/vcs/subprocessio.py

➞

Show inline comments

@@ @@ -310,97 +310,97 @@ class SubprocessIOChunker(object): @@
     Example usage:
     #    try:
     #        answer = SubprocessIOChunker(
     #            cmd,
     #            input,
     #            buffer_size = 65536,
     #            chunk_size = 4096
     #            )
     #    except (EnvironmentError) as e:
     #        print str(e)
     #        raise e
+    #
     #    return answer
     """
     def __init__(self, cmd, inputstream=None, buffer_size=65536,
                  chunk_size=4096, starting_values=None, **kwargs):
         """
         Initializes SubprocessIOChunker
         :param cmd: A Subprocess.Popen style "cmd". Can be string or array of strings
         :param inputstream: (Default: None) A file-like, string, or file pointer.
         :param buffer_size: (Default: 65536) A size of total buffer per stream in bytes.
         :param chunk_size: (Default: 4096) A max size of a chunk. Actual chunk may be smaller.
         :param starting_values: (Default: []) An array of strings to put in front of output que.
         """
         starting_values = starting_values or []
         if inputstream:
             input_streamer = StreamFeeder(inputstream)
             input_streamer.start()
             inputstream = input_streamer.output
         # Note: fragile cmd mangling has been removed for use in Kallithea
         assert isinstance(cmd, list), cmd
         _p = subprocess.Popen(cmd, bufsize=-1,
                               stdin=inputstream,
                               stdout=subprocess.PIPE,
                               stderr=subprocess.PIPE,
                               **kwargs)
         bg_out = BufferedGenerator(_p.stdout, buffer_size, chunk_size,
                                    starting_values)
         bg_err = BufferedGenerator(_p.stderr, 16000, 1, bottomless=True)
-        while not bg_out.done_reading and not bg_out.reading_paused and not bg_err.length:
         while not bg_out.done_reading and not bg_out.reading_paused:
             # doing this until we reach either end of file, or end of buffer.
             bg_out.data_added_event.wait(1)
             bg_out.data_added_event.clear()
         # at this point it's still ambiguous if we are done reading or just full buffer.
         # Either way, if error (returned by ended process, or implied based on
         # presence of stuff in stderr output) we error out.
         # Else, we are happy.
         _returncode = _p.poll()
         if _returncode or (_returncode is None and bg_err.length):
             try:
                 _p.terminate()
             except Exception:
                 pass
             bg_out.stop()
             out = ''.join(bg_out)
             bg_err.stop()
             err = ''.join(bg_err)
             if (err.strip() == 'fatal: The remote end hung up unexpectedly' and
                 out.startswith('0034shallow ')):
                 # hack inspired by https://github.com/schacon/grack/pull/7
                 bg_out = iter([out])
                 _p = None
             elif err:
                 raise EnvironmentError(
                     "Subprocess exited due to an error:\n" + err)
             else:
                 raise EnvironmentError(
                     "Subprocess exited with non 0 ret code:%s" % _returncode)
         self.process = _p
         self.output = bg_out
         self.error = bg_err
         self.inputstream = inputstream
     def __iter__(self):
         return self
     def next(self):
         if self.process and self.process.poll():
             err = ''.join(self.error)
             raise EnvironmentError("Subprocess exited due to an error:\n" + err)
         return self.output.next()
     def throw(self, type, value=None, traceback=None):
         if self.output.length or not self.output.done_reading:
             raise type(value)
     def close(self):

0 comments (0 inline, 0 general)