. Twisted理论基础

. 异步编程模式与Reactor

. Twisted网络编程

. reactor进程管理编程

. Twisted并发连接

. 在多线程程序中，对于停止某个线程启动另外一个线程，其决定权并不在程序员手里而在操作系统那里，因此，程序员在编写程序过程中必须要假设在任何时候一个线程都有可能被停止而启动另外一个线程

. 相反，在异步模型中，所有事件是以异步的方式到达的，然后CPU同样以异步的方式从Cache队列中取出事件进行处理，一个任务要想运行必须显式放弃当前运行的任务的控制权。这也是相比多线程模型来说，最简洁的地方 

. 在单线程同步模型中，任务按照顺序执行。如果某个任务因为I/O而阻塞，其他所有的任务都必须等待，直到它完成之后它们才能依次执行。这种明确的执行顺序和串行化处理的行为是很容易推断得出的。如果任务之间并没有互相依赖的关系，但仍然需要互相等待的话这就使得程序不必要的降低了运行速度 

. 在多线程版本中，这3个任务分别在独立的线程中执行。这些线程由操作系统来管理，在多处理器系统上可以并行处理，或者在单处理器系统上交错执行。这使得当某个线程阻塞在某个资源的同时其他线程得以继续执行。与完成类似功能的同步程序相比，这种方式更有效率，但程序员必须写代码来保护共享资源，防止其被多个线程同时访问。多线程程序更加难以推断，因为这类程序不得不通过线程同步机制如锁、可重入函数、线程局部存储或者其他机制来处理线程安全问题，如果实现不当就会导致出现微妙的bug 

. 有大量的任务，以至于可以认为在一个时刻至少有一个任务要运行

. 任务执行大量的I/O操作，这样同步模型就会在因为任务阻塞而浪费大量的时间

. 任务之间相互独立，以至于任务内部的交互很少

//这些条件大多在CS模式中的网络比较繁忙的服务器端出现(如WEB服务器)

https://likebeta.gitbooks.io/twisted-intro-cn/content/zh/p01.html

. 异步模式客户端一次性与全部服务器完成连接，而不像同步模式那样一次只连接一个，连接完成后等待新事件的到来

. 用来进行通信的Socket方法是非阻塞模的，这是通过调用setblocking()来实现的

. select模块中的select方法是用来识别其监视的socket是否有完成数据接收的，如果没有它就处于阻塞状态。

. 当从服务器中读取数据时，会尽量多地从Socket读取数据直到它阻塞为止，然后读下一个Socket接收的数据(如果有数据接收的话)。这意味着我们需要跟踪记录从不同服务器传送过来数据的接收情况 

. 监视一系列sockets(文件描述符)

. 并阻塞程序

. 直到至少有一个准备好的I/O操作 

. 监视一系列与I/O操作相关的文件描述符(description)

. 不停地汇报那些准备好的I/O操作的文件描述符

. 处理所有不同系统会出现的I/O事件

. 提供优雅的抽象来帮助在使用reactor时少花些心思去考虑它的存在

. 提供可以在抽象层外使用的公共协议实现 

while True:

    timeout = time_until_next_timed_event()

    events = wait_for_events(timeout)

    events += timed_events_until(now())

    for event in events:

        event.process()

. 网络和文件系统的限制

. 缓冲行为

. 如何检测连接丢失

. 出现错误时的返回值

. 一个是针对操作成功的回调

. 一个是针对操作失败的回调

. write: 以非阻塞的方式按顺序依次将数据写到物理连接上

. writeSequence: 将一个字符串列表写到物理连接上

. loseConnection: 将所有挂起的数据写入，然后关闭连接

. getPeer: 取得连接中对端的地址信息

. getHost: 取得连接中本端的地址信息

. makeConnection: 在transport对象和服务器之间建立一条连接

. connectionMade: 连接建立起来后调用

. dataReceived: 接收数据时调用

. connectionLost: 关闭连接时调用

https://likebeta.gitbooks.io/twisted-intro-cn/content/zh/p02.html

https://likebeta.gitbooks.io/twisted-intro-cn/content/zh/p04.html

http://blog.****.net/hanhuili/article/details/9389433

http://blog.sina.com.cn/s/blog_704b6af70100py9n.html

from twisted.internet.protocol import Factory

from twisted.protocols.basic import LineReceiver

from twisted.internet import reactor

class Chat(LineReceiver):

    def __init__(self, users):

        self.users = users

        self.name = None

        self.state = "GETNAME"

    def connectionMade(self):

        self.sendLine("What's your name?")

    def connectionLost(self, reason):

        if self.name in self.users:

            del self.users[self.name]

    def lineReceived(self, line):

        if self.state == "GETNAME":

            self.handle_GETNAME(line)

        else:

            self.handle_CHAT(line)

    def handle_GETNAME(self, name):

        if name in self.users:

            self.sendLine("Name taken, please choose another.")

            return

        self.sendLine("Welcome, %s!" % (name,))

        self.name = name

        self.users[name] = self

        self.state = "CHAT"

    def handle_CHAT(self, message):

        message = "<%s> %s" % (self.name, message)

        for name, protocol in self.users.iteritems():

            if protocol != self:

                protocol.sendLine(message)

class ChatFactory(Factory):

    def __init__(self):

        self.users = {} # maps user names to Chat instances

    def buildProtocol(self, addr):

        return Chat(self.users)

reactor.listenTCP(, ChatFactory())

reactor.run()

from twisted.internet import reactor

from twisted.internet.protocol import Protocol

from twisted.internet.endpoints import TCP4ClientEndpoint, connectProtocol

class Greeter(Protocol):

    def sendMessage(self, msg):

        self.transport.write("MESSAGE %s\n" % msg)

def gotProtocol(p):

    p.sendMessage("Hello")

    reactor.callLater(, p.sendMessage, "This is sent in a second")

    reactor.callLater(, p.transport.loseConnection)

point = TCP4ClientEndpoint(reactor, "localhost", )

d = connectProtocol(point, Greeter())

d.addCallback(gotProtocol)

reactor.run()

from twisted.internet import reactor

from twisted.internet.protocol import Protocol, ClientFactory

from sys import stdout

class Echo(Protocol):

    def dataReceived(self, data):

        stdout.write(data)

class EchoClientFactory(ClientFactory):

    def startedConnecting(self, connector):

        print 'Started to connect.'

    def buildProtocol(self, addr):

        print 'Connected.'

        return Echo()

    def clientConnectionLost(self, connector, reason):

        print 'Lost connection.  Reason:', reason

    def clientConnectionFailed(self, connector, reason):

        print 'Connection failed. Reason:', reason

reactor.connectTCP(host, port, EchoClientFactory())

reactor.run()

# Copyright (c) Twisted Matrix Laboratories.

# See LICENSE for details.

"""

An example IRC log bot - logs a channel's events to a file.

If someone says the bot's name in the channel followed by a ':',

e.g.

    <foo> logbot: hello!

the bot will reply:

    <logbot> foo: I am a log bot

Run this script with two arguments, the channel name the bot should

connect to, and file to log to, e.g.:

    $ python ircLogBot.py test test.log

will log channel #test to the file 'test.log'.

To run the script:

    $ python ircLogBot.py <channel> <file>

"""

# twisted imports

from twisted.words.protocols import irc

from twisted.internet import reactor, protocol

from twisted.python import log

# system imports

import time, sys

class MessageLogger:

    """

    An independent logger class (because separation of application

    and protocol logic is a good thing).

    """

    def __init__(self, file):

        self.file = file

    def log(self, message):

        """Write a message to the file."""

        timestamp = time.strftime("[%H:%M:%S]", time.localtime(time.time()))

        self.file.write('%s %s\n' % (timestamp, message))

        self.file.flush()

    def close(self):

        self.file.close()

class LogBot(irc.IRCClient):

    """A logging IRC bot."""

    nickname = "twistedbot"

    def connectionMade(self):

        irc.IRCClient.connectionMade(self)

        self.logger = MessageLogger(open(self.factory.filename, "a"))

        self.logger.log("[connected at %s]" %

                        time.asctime(time.localtime(time.time())))

    def connectionLost(self, reason):

        irc.IRCClient.connectionLost(self, reason)

        self.logger.log("[disconnected at %s]" %

                        time.asctime(time.localtime(time.time())))

        self.logger.close()

    # callbacks for events

    def signedOn(self):

        """Called when bot has succesfully signed on to server."""

        self.join(self.factory.channel)

    def joined(self, channel):

        """This will get called when the bot joins the channel."""

        self.logger.log("[I have joined %s]" % channel)

    def privmsg(self, user, channel, msg):

        """This will get called when the bot receives a message."""

        user = user.split('!', )[]

        self.logger.log("<%s> %s" % (user, msg))

        # Check to see if they're sending me a private message

        if channel == self.nickname:

            msg = "It isn't nice to whisper!  Play nice with the group."

            self.msg(user, msg)

            return

        # Otherwise check to see if it is a message directed at me

        if msg.startswith(self.nickname + ":"):

            msg = "%s: I am a log bot" % user

            self.msg(channel, msg)

            self.logger.log("<%s> %s" % (self.nickname, msg))

    def action(self, user, channel, msg):

        """This will get called when the bot sees someone do an action."""

        user = user.split('!', )[]

        self.logger.log("* %s %s" % (user, msg))

    # irc callbacks

    def irc_NICK(self, prefix, params):

        """Called when an IRC user changes their nickname."""

        old_nick = prefix.split('!')[]

        new_nick = params[]

        self.logger.log("%s is now known as %s" % (old_nick, new_nick))

    # For fun, override the method that determines how a nickname is changed on

    # collisions. The default method appends an underscore.

    def alterCollidedNick(self, nickname):

        """

        Generate an altered version of a nickname that caused a collision in an

        effort to create an unused related name for subsequent registration.

        """

        return nickname + '^'

class LogBotFactory(protocol.ClientFactory):

    """A factory for LogBots.

    A new protocol instance will be created each time we connect to the server.

    """

    def __init__(self, channel, filename):

        self.channel = channel

        self.filename = filename

    def buildProtocol(self, addr):

        p = LogBot()

        p.factory = self

        return p

    def clientConnectionLost(self, connector, reason):

        """If we get disconnected, reconnect to server."""

        connector.connect()

    def clientConnectionFailed(self, connector, reason):

        print "connection failed:", reason

        reactor.stop()

if __name__ == '__main__':

    # initialize logging

    log.startLogging(sys.stdout)

    # create factory protocol and application

    f = LogBotFactory(sys.argv[], sys.argv[])

    # connect factory to this host and port

    reactor.connectTCP("irc.freenode.net", , f)

    # run bot

    reactor.run()

class LogBotFactory(protocol.ClientFactory):

    protocol = LogBot

    def __init__(self, channel, filename):

        self.channel = channel

        self.filename = filename

http://twisted.readthedocs.org/en/latest/core/howto/clients.html

#!/usr/bin/env python

# Copyright (c) Twisted Matrix Laboratories.

# See LICENSE for details.

from twisted.internet import protocol

from twisted.internet import reactor

import re

class MyPP(protocol.ProcessProtocol):

    def __init__(self, verses):

        self.verses = verses

        self.data = ""

    def connectionMade(self):

        print "connectionMade!"

        for i in range(self.verses):

            self.transport.write("Aleph-null bottles of beer on the wall,\n" +

                                 "Aleph-null bottles of beer,\n" +

                                 "Take one down and pass it around,\n" +

                                 "Aleph-null bottles of beer on the wall.\n")

        self.transport.closeStdin() # tell them we're done

    def outReceived(self, data):

        print "outReceived! with %d bytes!" % len(data)

        self.data = self.data + data

    def errReceived(self, data):

        print "errReceived! with %d bytes!" % len(data)

    def inConnectionLost(self):

        print "inConnectionLost! stdin is closed! (we probably did it)"

    def outConnectionLost(self):

        print "outConnectionLost! The child closed their stdout!"

        # now is the time to examine what they wrote

        #print "I saw them write:", self.data

        (dummy, lines, words, chars, file) = re.split(r'\s+', self.data)

        print "I saw %s lines" % lines

    def errConnectionLost(self):

        print "errConnectionLost! The child closed their stderr."

    def processExited(self, reason):

        print "processExited, status %d" % (reason.value.exitCode,)

    def processEnded(self, reason):

        print "processEnded, status %d" % (reason.value.exitCode,)

        print "quitting"

        reactor.stop()

pp = MyPP()

reactor.spawnProcess(pp, "wc", ["wc"], {})

reactor.run()

class GPGProtocol(ProcessProtocol):

    def __init__(self, crypttext):

        self.crypttext = crypttext

        self.plaintext = ""

        self.status = ""

    def connectionMade(self):

        self.transport.writeToChild(, self.passphrase)

        self.transport.closeChildFD()

        self.transport.writeToChild(, self.crypttext)

        self.transport.closeChildFD()

    def childDataReceived(self, childFD, data):

        if childFD == : self.plaintext += data

        if childFD == : self.status += data

    def processEnded(self, status):

        rc = status.value.exitCode

        if rc == :

            self.deferred.callback(self)

        else:

            self.deferred.errback(rc)

def decrypt(crypttext):

    gp = GPGProtocol(crypttext)

    gp.deferred = Deferred()

    cmd = ["gpg", "--decrypt", "--passphrase-fd", "", "--status-fd", "",

           "--batch"]

    p = reactor.spawnProcess(gp, cmd[], cmd, env=None,

                             childFDs={:"w", :"r", :, :"w", :"r"})

    return gp.deferred

http://twistedmatrix.com/documents/12.2.0/core/howto/process.html

. Eventlet (python)

. Gevent (python) is similar to eventlet uses libevent which is build on top of epoll.

. C++ ACE

. Java Netty

. Ruby Eventmachine

Poll for Linux

Epoll for Linux 2.6

WaitForMultipleObjects (WFMO) for Win32

Input/Output Completion Port (IOCP) for Win32

KQueue for FreeBSD and Mac OS X

CoreFoundation for Mac OS X

from twisted.internet import selectreactor

selectreactor.install()

from twisted.internet import reactor

from twisted.internet import pollreactor

pollreactor.install()

from twisted.internet import reactor

from twisted.internet import kqreactor

kqreactor.install()

from twisted.internet import reactor

from twisted.internet import win32eventreactor

win32eventreactor.install()

from twisted.internet import reactor

from twisted.internet import iocpreactor

iocpreactor.install()

from twisted.internet import reactor

from twisted.internet import epollreactor

epollreactor.install()

from twisted.internet import reactor

https://moythreads.com/wordpress/2009/12/22/select-system-call-limitation/

http://pipeit.blogspot.com/2011/07/select-poll-and-epoll-twisted-story.html

http://twistedmatrix.com/documents/13.2.0/core/howto/choosing-reactor.html#auto2