1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
| #Docs {{{1
'''
__name__ = net_talk
__author__ = N.tox
__version__ = 1.1
net_talk module aim to facilitate the design process of a coded language
beetween two computer for networking.
It has nothing to do with protocols.
Often, when clients and server communicate, as they can only exchange text,
there is a need to use the first(s) octet(s) to represent an instruction,
request, answer, information, etc... We will call these bytecodes.
What those bytecodes represent are totally arbitrary, but the set of these
represents a sort of language by itself, and designing it can be tough, or at
least boring and long. The readability is an important factor in the design
process' speed. The maintainability of a such "language" can quickly become a
hell if not handled properly.
This is where this module can be usefull :
- the design process is very quick, because of a high readability
- the bytecodes (arbitrary octets) are automatically assigned
- the message encoding and decoding is very easy, and readable (cause you
don't personnaly have to read, interpret or navigate through the
bytecodes)
- the language tree can be dynamicaly modified (you must be careful though)
Quick Workflow Overview : {{{2
the following are big lines of the order to follow:
- create a string that represents all the kind of messages that will be
exchanged beetween the client and the server (or only client, or only
server, or anything else, your call), this is the model of the language
tree.
- pass this model to the "build" function, which will return a tree of the
language
- call a tail node of this tree to get a full message, ready to be sent
over the network
- when receiving a message, pass this message to the method "handle" of the
tree for it to call the custom handler attached to the corresponding node
(understand that you don't have to find the node). Alternatively, you can
call the method tree.find(received_message) which will return a 2 tuple
of the corresponding node, and the data part of the message (the part
after the bytecodes).
Model rules and usage : {{{2
A Model is a string constituted of multiple lines, it can have only one,
but in this case I recommend you to not even bother with this module.
elements to be aware of : {{{3o
to make a model, you must dinstinguish few elements essentials in its
redaction:
- the ident_char
- the sep_char
- the bytecode_size
- the name
- the data_encoder_name
- the handler_name
ident_char : {{{4
The function used to parse the model bases its parenting logic on the
identation.
The ident_char is an optional function argument.
The ident_char MUST be a string
The default ident_char is four spaces ' '
The ident_char MUST NOT be a single space
The ident_char CAN be any non-digit, non-alpha character(s), you want
but NOT '_', '\\n'
The ident_char MUST NOT be the same as sep_char
sep_char : {{{4
The sep_char is a character used to separate the name, the
data_encoder_name and the handler_nam
The sep_char is an optional function argument
The sep_char default is ':'
The sep_char SHOULD be a single character
The sep_char MUST NOT be alpha, digit, '\\n', '_', ident_char
The sep_char MUST NOT be preceeded or followed by spaces
bytecode_size : {{{4
The bytecode_size inside a model, most of the time SHOULD BE IGNORED,
unless you know that the node's children number will be over 255 (1
byte), in which case you can specify a size of 2 bytes (meaning, a
maximum of 65535 children for the node). Specify a bytecode_size > 2 is
insane if justified, otherwise uterly useless and a waste of bytes.
If used, the bytecode_size must be specified RIGHT AFTER the
identation and BEFORE the name. For readability, it is recommended that
a single space separate the bytecode_size from the name
The bytecode_size MUST be one (or more) digit
name : {{{4
The name indicate the name of the node, plain and simple
The name obey to the python naming rules, just a matter of common sense
The name MUST be specified RIGHT AFTER the identation or the
bytecode_size
the name MUST be specified BEFORE the data_encoder_name
data_encoder_name: {{{4
The data_encoder_name indicate the name of the callable object which
will be associated to the node, the goal of a such object is to encode
its arguments into a string the way you see fit. As long as it return
a string, it can does whatever you want.
The data_encoder_name is PURELY OPTIONAL
IF SPECIFIED, the node then CANNOT have any children. Therefore,
specifying the bytecode_size is non-sens.
IF SPECIFIED, it MUST be done AFTER the name, and BEFORE the
handler_name
The data_encoder_name obey to python naming rules
handler_name : {{{4
The handler_name indicate the name of the callable object which will be
associated to the node. The goal of a such object is generaly to
decode the string produced by the data_encoder. But it can does
whatever you want.
The handler_name is PURELY OPTIONAL
IF SPECIFIED, the node then CANNOT have any children. Therefore,
specifying the bytecode_size is non-sens.
IF SPECIFIED, it MUST be done AFTER the data_encoder_name OR after two
sep_char (in this case, data_encoder_name must not be specified)
The handler_name obey to python naming rules
example : {{{3
client
message
private:private_message_encoder:private_message_handler
public::public_message_handler
quit
2 server
message
warning
kick::kick_handler
Tree structure : {{{2
A language tree is composed up to three objects:
- 1 MsgRoot object
- 0+ MsgNode objects
- 1+ MsgTail objects
Inside a model, only <MsgNode> and <MsgTail> are implicitly represented ;
the <MsgRoot> - as its name suggest - just serves as a base for the
structure and doesn't have any bytecode.
As stated above, the <MsgRoot> is the base of the tree. There can only be
one per tree, obviously.
<MsgNode> just fullfill an intermediate role, to provide... a node :p. It
possess a bytecode though. In most cases, you shouldn't worry about this
lats information.
<MsgTail> are well, the end of a branch. They possess a bytecode too, and
in a raw message, they are the last bytecode before proper data to be dealt
with. They are by far, the most important. Indeed, their strengh comes
essentially from the fact that we can associate to them two callable
objects : one to converts some python object into a string, and one to
decode this very string back into data (and eventually directly process
it). This offers essentialy to the programmer more readable source code and
a higher maintainability.
Workflows : {{{2
there are several workflows possible, each will be exposed here through
examples, but always with quite the same language tree.
Quickest way : {{{3
The quickest way, consists in creating a module which only purpose is
to create the language tree and all the needed callables. Then, import
the created the tree into the client and server.
[comm.py]
from net_talk import *
model = """
client
message
public:public_encoder:public_handler
private:private_encoder:private_handler
quit::quit_handler"""
def public_encoder(message):
assert isinstance(message,str)
return message
def public_handler(data):
return data
def private_encoder(to, message):
assert isinstance(to, str)
assert isinstance(message, str)
return encode_data([to, message])
def private_handler(data):
return extract_data(data)
tree = build(model)
[/comm.py]
Now, comm.tree can be imported into the client and server modules and
be used.
Now, the client can call tree.client.message.private('john','fooooooooooo')
to obtain a string ready to be sent over the network.
Then the server receive the message, and call tree.find(message) which
will return the concerned node and the data part of the raw message.
But there is major weaknesses, in this methodology, because the
receiver of the message, will still have to identify the nature of the
node somehow in order to handle properly the decoded data provided. This is
why this metology is NOT RECOMMENDED in most cases. Though, some
program's architecture make this metodology flawless.
Efficient : {{{3
The most efficient methodology is to create a module in which you just
build the language, whithout any handlers, you can still define
data_encoders in it, but this is optional and not recommended.
Then, importing built tree into the server and client modules, and
assigning from them some handlers and/or data_encoders to concerned
<MsgTail>, regarding the job of the module.
Step 1 - creating comm.py :
[comm.py]
from net_talk import *
model="""
client
message
private
public
server
message"""
tree = build(model)
[/comm.py]
Step 2 - Inside client.py
[client.py]
*** some imports ***
from comm import tree
from net_talk import extract_data, encode_data
message = tree.client.message #shortcut
sr_msg = tree.server.message #shortcut
@message.public._command_deco
def public_message(msg):
assert isinstance(msg,str)
return encode_data(msg)
@message.private._command_deco
def private_message(reciever, message):
assert isinstance(reciever, str)
assert isinstance(message, str)
return encode_data([reciever, message])
@sr_msg._handler_deco
def server_message(data):
msg = extract_data(data)
text_widget.append(msg)
*** some code ***
def on_key_enter():
msg = entry_widget.get_message()
reciever = get_reciever()
if reciever :
connection.send( message.private(reciever, msg) )
else:
connection.send( message.public(msg) )
[/client.py]
Step 3 - Inside server.py
[server.py]
*** some imports ***
from comm import tree
from net_talk import extract_data, encode_data
cl_msg = tree.client.message #shortcut
sr_msg = tree.server.message #shortcu
@cl_msg.private._handler_deco
def private_message(data):
reciever, msg = extract_data(data)
connections[reciever].send( tree.server.message(msg) )
@cl_msg.public._handler_deco
def public_message(data):
msg = extract_data(data)
for connection in connections:
connection.send( tree.server.message(msg) )
@sr_msg._command_deco
def _message(msg):
assert isinstance(msg, str)
return encode_data(msg)
*** some code ***
msg=connection.recieve()
tree.handle(msg)
[/server.py]
'''
import string as _string
__all__ = ('MsgRoot', 'build', 'extract_data', 'encode_data')
LEGAL_CHAR = _string.ascii_letters+'_'
#Private {{{1
#Classes {{{2
class _MsgRoot_Build_Looper(object):
def __init__(s,text):
s._data = text
def _first_line_idx(s):
return s._data.find('\n')
def next(s):
idx = s._first_line_idx()
if idx == -1:
return s._data
else:
return s._data[:idx]
def __iter__(s):
while 1:
if not s._data: raise StopIteration
yield s.next()
def pop(s):
idx=s._first_line_idx()
if idx==-1:
d,s._data=s._data,''
else:
d,s._data=s._data[:idx],s._data[idx+1:]
return d
def __nonzero__(s):
return bool(s._data)
#Functions {{{2
def _byted_size(string, num_bytes=2):
val = len(string)
return ''.join(chr((val<<(8*i))&0xff) for i in xrange(num_bytes))
def _unbyte_size(byted_size):
return sum(ord(c)>>(8*i) for i,c in enumerate(byted_size))
#Public {{{1
class MsgRoot(object):#{{{2
#private {{{4o
def __init__(s,bytecode_size=1):
assert isinstance(bytecode_size,int)
assert 0<bytecode_size<256
s._childs = 0
s._set_bytecode_size(bytecode_size)
def __iter__(s):
for key in s.__dict__:
if key[0] != '_':
yield getattr(s,key)
def __setattr__(s,key,val):
assert key[0] == '_'
object.__setattr__(s,key,val)
def _set_bytecode_size(s,v):
s._bc_sz = v
s._max = sum(0xff<<(8*x) for x in xrange(s._bc_sz))
def _attribute_id(s):
r = s._childs
if r > s._max:
raise Exception('Maximum number of childs reached')
s._childs += 1
return ''.join(chr((r>>(8*i))&0xff) for i in xrange(s._bc_sz))
def _build(s,lines,parent,ident_char,sep_char,_ident):
assert isinstance(parent,MsgRoot)
assert isinstance(_ident,int)
def nest_get_bytecode_size(string):
for i,c in enumerate(string):
if c in LEGAL_CHAR:
break
if i in (0,len(string)-1):
return 1,string
bc_sz = string[:i].strip()
if not all(map(str.isdigit,bc_sz)):
raise Exception('Invalid syntax in your model, line :\n\t"%s"'%'{}{}'.format(ident_char*_ident,string))
return int(string[:i].strip()),string[i:]
def nest_strip(string):
return string.replace(ident_char,'').strip()
for line in lines:
if not nest_strip(line):
lines.pop()
continue
ident = line.count(ident_char)
if ident == _ident:
line = line.replace(ident_char,'')
bytecode_size,line = nest_get_bytecode_size(line)
args = line.split(sep_char)
assert 0<len(args)<4
assert args[0][0] in LEGAL_CHAR
if len(args) == 1: #means MsgNode
parent.add_child(args[0], bytecode_size=bytecode_size)
elif len(args) == 2: #means MsgTail
name,command = args
if command:
assert command[0] in LEGAL_CHAR
parent.add_child(name, command=eval(args[1]), bytecode_size=bytecode_size)
else:
parent.add_child(name, command=None)
elif len(args) == 3: #means MsgTail with handler
name,command,handler = args
if command:
assert command[0] in LEGAL_CHAR
command = eval(command)
else:
command = None
if handler:
assert handler[0] in LEGAL_CHAR
parent.add_child(name, command, eval(handler), bytecode_size=bytecode_size)
else:
parent.add_child(name, command, bytecode_size=bytecode_size)
last = getattr(parent,args[0])
lines.pop()
elif ident>_ident:
if 'last' not in locals(): raise Exception('Please, recheck your model identation or function\'s arg "_ident"')
s._build(lines, last, ident_char, sep_char, ident)
else:
return
def _set_tails(s):
for node in s:
if not node._childs:
node.tail_transmute()
else:
node._set_tails()
#public {{{4o
def add_child(s, name, command=None, handler=None, bytecode_size=1):
'''<Node>.add_child(name, command=None, handler=None, bytecode_size=1) --> None
name <-- str : name of the child (obey to python naming rules)
command <-- callable : callable object that must return a string
handler <-- callable : callable object that have only one argument (self
argument in cases of classes instances is not taken in account)
Please, refer to the module docs for further details'''
assert name
assert isinstance(name,str)
assert name not in s.__dict__
if command is not None or handler is not None:
object.__setattr__(s, name, MsgTail(bytecode_size, s._attribute_id(), s, command, handler))
else:
object.__setattr__(s, name, MsgNode(bytecode_size, s._attribute_id(), s))
def is_tail(s):
return isinstance(s ,MsgTail)
def tail_transmute(s):
raise Exception('MsgRoot objects cannot become tails')
def find(s, string):
'''<MsgRoot>.find(string) -> (<MsgTail>, data)
string <- str : must be a raw message received from network
search throug the tree the coresponding <MsgTail>, and
return it along with the data (part of the string after the
bytecodes)'''
for node in s:
if string.startswith(str(node)):
if node.is_tail():
return node,string[len(str(node)):]
return node.find(string)
raise Exception('string [%s] does not coreespond to any tail of the language'%repr(string))
def handle(s, string):
'''<MsgRoot>.handle(string) -> ???
string <- str : must be a raw message received from network
Is the same as :
>>> node,data = <MsgRoot>.find(raw_message)
>>> node.handle(data)'''
node,arg = s.find(string)
return node.handle(arg)
def build(s,text,ident_char=' '*4, sep_char=':',_ident=0):
assert all(isinstance(x,str) for x in (ident_char,sep_char))
assert ident_char != sep_char
assert ident_char not in sep_char
s._build(_MsgRoot_Build_Looper(text),s,ident_char,sep_char,_ident)
s._set_tails()
class MsgNode(MsgRoot): #{{{2
def __init__(s,bytecode_size,code,parent):
MsgRoot.__init__(s,bytecode_size)
s._bytecode = code
s._parent = parent
def __str__(s):
if isinstance(s._parent,MsgNode):
return str(s._parent)+s._bytecode
return s._bytecode
def __len__(s):
return len(str(s))
def tail_transmute(s):
assert not s._childs
s.__class__ = MsgTail
s.__init__(s._bc_sz,s._bytecode,s._parent)
class MsgTail(MsgNode): #{{{2
@staticmethod
def _dummy(*args,**kwargs): return ''
def _command_deco(s, _callable):
def wrapper(*args,**kwargs):
return _callable(*args,**kwargs)
s._command = wrapper
def _handler_deco(s, _callable):
def wrapper(*args,**kwargs):
return _callable(*args,**kwargs)
s._handler = wrapper
def __init__(s,bytecode_size,code,parent,command=None,handler=None):
MsgNode.__init__(s,bytecode_size,code,parent)
if command is None:
command = s._dummy
if handler is None:
handler = s._dummy
assert callable(command)
assert callable(handler)
s._command=command
s._handler=handler
def __call__(s,*args,**kwargs):
'''<MsgTail>.__call__(s,*args,**kwargs) -> str
call the associated command, which MUST return a string'''
r=s._command(*args,**kwargs)
assert isinstance(r,str)
return str(s)+r
def set_command(s,command):
assert callable(command)
s._command=command
def set_handler(s,handler):
assert callable(handler)
s._handler=handler
def handle(s,data):
'''<MsgTail>.handle(data) -> ???
data <-- str : is the part of the raw message after the bytecodes
call the associated handler, wich can return what you want.
Generaly, the goal of the handler is to decode the data, and return it,
but it can directly process it, or do whatever you want. Your call.'''
return s._handler(data)
def add_child(s):
raise Exception('MsgTail objects cannot have any childs')
def tail_transmute(s):
pass
def build(model, bytecode_size=1): #{{{2
'''build(model, bytecode_size=1) -> MsgRoot
model <-- string representing the tree language
bytecode_size <-- int indicate the size (in bytes) of the bycodes attributed
to the MsgRoot's direct children'''
tree=MsgRoot(bytecode_size)
tree.build(model)
return tree
def encode_data(iterable, size_num_bytes=2): #{{{2
if not hasattr(iterable, '__iter__'):
iterable = [iterable]
assert all(isinstance(x, str) for x in iterable)
return ''.join(_byted_size(x,size_num_bytes) for x in iterable)
def extract_data(data, size_num_bytes=2, _data=None): #{{{2
sb = size_num_bytes
if _data is None:
_data = []
size = _unbyte_size(data[:sb])
data_end = sb+size
_data.append(data[sb:data_end])
data = data[data_end:]
if data:
extract_data(data, sb, _data)
if len(_data) == 1:
return _data[0]
return _data |
[net_talk.py] avis, corrections, retours
Répondre avec citation
Partager